I imagine him to be somewhere near the castle but not actually inside them if I am seeing this place clearly.

He just saw Aurora though so I suppose somewhere in the streets.