[quote=XecutionerRex]
Going by that logic, it would seem alright to tell someone they're ugly to their face, because I "observed" that.
[/quote]

Well yes actually. Ugly isn't necessarily an insult. It depends on the culture in which you are pressured by. 

Now ugly in most cultures is an insult, but if I do remember correctly there is a British magazine that uses so called 'Ugly' models for their pictures and advertisement. 

Insults are all based on what people think. 

Observations are all based on what the thing is.