If they have to destroy drones, they'll suspect that someone's a spy and move you.  The ex-CIA person would be an obvious suspect.
I'd just say that you were under deep cover, and they lost contact.  This is only going to last a couple of weeks in-game (I may only do one week of torture before it gets burned to the ground.), and I'm not sure being out of contact for a week will mean that they storm it, as they don't want to break your cover.