As I recall, the rules say the town's population is drawn from the fief in which it is located. Only if the town population exceeds fief population may you draw population from another fief.
Flooby, rather than leave my vassals choice, would it be possible for you replicate and divide the army I posted amongst the vassals? It may not divide evenly in 3 though, but I would not begrudge rounding.