As I recall, the rules say the town's population is drawn from the fief in which it is located. Only if the town population exceeds fief population may you draw population from another fief.