Doesn't have to be banished, just sent off somewhere...because no, I don't think he can without your permission