I guess silver guards find you in the village and bring you back to the castle