[quote=Xaxl]
And by invading/pillaging other factions with an organized military force you are inviting yourself to be completely wiped out as an aggressively xenophobic warmonger.
[/quote]

Pretty much this, and the larger your faction is, the easier it will be for a larger and more powerful nation/conclave of nations to find and destroy you.