My faction cannot work conceptually without shitloads of territory. If there isn't many inhabited worlds there aren't many places churning out bodies. If I'm to far away the time it takes to conquer anything would be far too long to allow me to actually interact with anyone, and third if there isn't much 'land' then what has the Empire been doing? jacking off? 

No, without the territory I can't have my faction, therefore instead of arguing and arguing and arguing for something I won't be able to have I'll simply withdraw to prevent any arguing or frustrations.