Well my thought was that he fled the giantkin due to poor treatment as the runt in the litter, and while he absolutely still is a magical creature, he can fully pass as just an enormous human, which makes him sympathetic enough to not garner outright hostility. I'm really just imagining a brutish eight-foot-tall man,  sort of like if you scaled Andre the Giant up another 10%.