I think it'd be a bit easier, especially in terms of starting off, to allow imps in some way.

maybe the number of imps you control is determined by the number of human, slaves or otherwise, under you, and starting with a set maximum. Say you can start off with 5 imps max, in order to summon more you need to enslave, convert, etc, humans, perhaps an imp for every 4-5 humans. Think of it as like the imps are taskmasters over groups of humans.

Just an idea. Otherwise, I like what you have done so far.