Rilla, and his family, the Sacred, weren't exactly the nicest people. In fact, they were all pretty much assholes. Rilla's family, especially, is almost always warring with each other.

No need to oppose him now though, he's basically dead. Ouroboros has taken a new person, Rilla's grandson, Joseph, the Imaginist. He's actually pretty much worse, so they may end up opposing him.