You've got the main jist of it. The Robotech masters were Tirolian, and sort of an arrogant and selfish 'ruling class'. They left with the people they decided they wanted with the - their soldiers and various other people they decided were worthwhile - and abandoned the rest of the population on Tirol as they had no use for them. For whatever reason, they didn't meet their 'standards'. 
They're all that's left behind, and as far as the Regent is concerned, they're just as deserving of blame as the Masters.