There are a few ways you could go about it.

One way, that would expand on the concept of the oracles as wise and clever, is to say that they were aware of the problems facing the Empire and could see it's coming doom through social patterns, so they did little things to help that doom along. There were in the background whispering the wrong advice into the right ears, and using their position in the libraries to manipulate the flow of information. This would be similar to what you already have, but it give the Visha history a little more breathing room since they would be exacerbating issues rather than straight-out causing them.

Another option is giving that part of their backstory to the uncertainty of history. Some oracles claim that their race brought down the Visha. Others claim that they used their knowledge to protect themselves from the effects of the fall and had no reason to bring it down. When doing things like this, historical obfuscation can be useful.