Because a game with as much content at Skyrim really needed to be lagged out by a sea of meaningless NPCs every time you approached one of the holds.