I have an idea I figured out.

How about when we exit the mirror world we end up in the abandoned building, with the big mirror and all. In other words our meeting place will also be our primary entry point. We can't use the house of mirrors, since too many mirrors, throws you to different parts of the town at almost random, therefore using the common entry point would be safer.

They used that logic for the TV section in Persona 4 as well.