Before I finish my post, what if Kaoru knocks on the swordsmen compound and they're not particularly happy to see him? Perhaps they try to beat him up or something but he's saved by his teammates, who work together for once? Just an idea, I can always have him run away and lose them.