Not to mention the problems that arise in displacing large groups of people anyway.

Or that everyone and their grandma use magic in Earthland, and a bunch of magic-less people would be easy target for magical bandit gangs.