So you're ungodly fast, you can warp in and out quite quickly, you've got good, strong armor, you've got supersmart swarmer missiles...

So what are the cons? Besides them each being loners? Cause you're saying that you're really fast, you're hard to kill, you're hard to avoid getting killed by, and you can FTL in and out quickly. What balances it? Making them each loners is hardly a balancing act. They are smaller craft, I assume, and can be have a crew trained and ready to operate very very quickly.

Really, if something threatens, then your peoples could just band together out of fear for extinction, and it'd be impossible to beat you. Not without some really powerful laser weapons or other instantaneous types to disable before you can run away.