[@The Harbinger of Ferocity] While broadly you are correct the difference I am talking about is literally height alone. Yes a 5'10 guy might appear slightly more capable of defending himself than 5'6 one, but that ignores all the variables you just stated of build, dress, expression, posture etc. And this is still observing the hypothetical in the 'fair' 1 on 1 context. A 5'10 guy could be skinny and scared looking and a 5'6 guy could be strong, stocky and mean looking, so my challenge is to the woman, who would you feel more 'protected' by? 

And of course my original point still stands, a a lone person's few inches in height will not deter a group of assailants when they know they have the number advantage.