Posted for the Caravan mission. You can either engage the bandits now or wait for them to leave, lessening the numbers. And yes, they will be back before long, and with the other bandits who went hunting. So do whatever.