[quote=@Nimda]
<Snipped quote by Audacious>

Sure, but they don’t always gang together. And if you have this secret society, couldn’t you oust them or something?
[/quote]

It's a little deeper than that. They genuinely don't always gang together, but it's not like everybody knows everybody either, and not everyone has a stake in every problem.