Well, look at it this way- mages can't be told at a glance from normal people. Guardians are probably the same. So, Mr feral mage sees what he takes to be a nobody, chucks a fireball at you, and suddenly bad for him.