1. Install the 'Inconsequential NPCs' mod, and clear this right up. Supplement with 'Populated roads,' and other such population mods. Suddenly you need a party of ten to clear a fortress -- and that's awesome.

2. ESO revisited Skyrim (albeit in an earlier age) and did a much, MUCH better job of filling it up with NPCs.