Maybe the raiders siege the village? force the outsiders to stay inside? Maybe the raiders threaten to return, and the outsiders, either through personal morals or promise of a paycheck by the mayor agree to stay and defend the village?