You do have a valid point. Perhaps there could be several different bandit raids, after all our clan it is impossible to avoid bandits, but I don't believe that there would be enough bandits to support three different raids. 

Maybe Mardox's character could have been selected after everyone got settled down in the Hidden Valley and the Warriors wanted him on the ring. However, we could also pushed the bandit raid to when they did get settled in the Hidden Valley. As for your character, perhaps the the age be scaled up slightly to fit. However, if you don't wish to change that part of history, perhaps the a a rival clan attacked ours before our true exile.

Any of these tickling your fancy?