The bandits are coming out of the wall so you'd actually have to be pretty close to the wall to be coming out there.

Would be better if you started near the plaza. The main western street is where Branna, Thom, and Alnard are, which connects to the plaza.