[@AimeChambers]
He could be upset about being largely raised by servants, as well as the loss of his mother. Perhaps he believes that King Vincent could have done more to protect his mother, and even resent him for kicking her out in the first place. Also, it wouldn't be a surprise if the nobles, servants, and tutors commented on his heritage to him when the "pure" royal family wasn't around.