Yes you can but the Magi's place is Avalon, they will very rarely leave it. Also, you might be forced to attack it sooner or later and that would ruin all chances of an alliance!