I'd also normally suggest killing the guys inside when the others have left so we can use their own fortifications against them, but I'm not sure we have time.