It wouldn't be unusual for a lord to live in a mansion at a more secure city, and then he just sends goons out to his villages to collect his tithe. Especially if the minerals at the village is as valuable as Kath and Ashara were told, the man could probably afford to live so far away. None of the militia there seem particularly loyal to him, so it's not like the guy stations his own goons to even protect the place. Just goes to show how disconnected he is from his own people.