The guards do nothing to stop you departing the keep, but do seem to return to their bickering among one another.  This time, they do it in a much more hushed fashion. shooting the occasional glance at the departing caravan guards as if still not completely certain about their intentions.  The sun is setting rapidly and it will be dark very soon.