All of the challengers face the gyms in the same order, which is why there is always a crowd of people waiting for you; they are there to watch the challenge take place. It's also why the first three leaders all show up to see you off once you beat Kabu; because most challengers don't make it past the third gym. Kabu is the first big hurdle and most challengers fail at that point.

I was trying to find a source that says the order is based on strength, but I can't find one. Maybe I'm making that up.

But they do also have multiple teams depending on how many badges the challengers have, it's just that they know exactly where in the order they will sit.