Support would grow rapidly, though. The only thing that's needed is to have the majority of the army out in the open, away from the castle. Perhaps traveling to another kingdom for more slaves and to destroy and loot it.

The way I would go about it is getting the army to cross a plain between two hilly, forested areas. Perhaps a major trade route, where it had been deliberately widened for the purpose of large groups of people passing each other easily, and so they can camp on the edges without having to find a forest clearing, etc etc etc, which would give horsemen room to manuever, and archers and the like to effectively hide and snipe from the trees. Then it'd be a relatively easy manner in finding enough troops for several lines to block a narrow point, so that they can force the army to come at them at much more manageable numbers, so that archers can pick them off and calvary can strike at their rear ranks should they try to retreat.