Asides from the Alliance, there's the Dragonier. Before the current circumstances, most of the nations' rulers or at least one of their relatives were part of the Dragonier at some point and would've formed close bonds while working together to keep the peace throughout all of Tirn.

For example, the current Shokhan of Yolta is a nephew of Grandmaster Temuro and Sifa personally knew the ancestors of each nations' rulers. Kasimir himself made many friends during his time among the Dragonier and some of them now serve as his most elite warriors and generals.

Because of their history, a good portion of the Dragonier were usually from Kovosia. Almost half I'd say, so it's a fair bet to assume that most of them have gone over to the Empire by now. The only ones I can see staying behind are either commoners who don't have a reason to go back or nobles holding a personal grudge against Kasimir. Maybe they lost relatives during his rise to power or their houses were overthrown during the unification of Kovosia.

To be sure, there'd be a few Norsinians who actually support the Empire's cause but with the passes through the Stormwall Mountains being heavily guarded, none would be able to openly switch sides atm. As for the elven houses... Yeah, they're just straight up not having a good time right now. Absolutely none of them are going to side with Kasimir. Not willingly anyway. 

Sorry for the lengthy response, I didn't really know what you meant by 'houses' so I just interpreted it broadly and tried to cover everything.