It's not an unfair assumption either way. Castle's typically encapsulate at least a small village along with the main palace. Importing food during these times would be expensive and inefficient so you would have a village of workers who work the farms, guards and the families thereof. Get enough people to ensure that you can feed everyone and it's safe to assume you'd build a place of worship (though we haven't established any religion left) where they can all go. If this is feudal as it appears it is House Tosen would be like a Lord serving a king, they would have power over a large amount of land which they would "lease" out to other nobles or to townsfolk to grow their food.