Maybe that was the enemy's plan. :P Send out a double to trick them into going to the keep, and if that fails, force them.