Generally speaking in realistic or semi-realistic settings everything is a trade-off. You might be able to scrape your plucky witch character into feudal combat if she's poorly armoured or outmatched in direct melee, you wouldn't be able to scrape a fully armoured swordsman who also has magic into the same setting. You're covering too many bases, and it pops your character up into the next category of power as a result. 

Taking magic out of the equation, going unarmoured is generally a bad idea, but it does have its advantages. Armour limits mobility, particularly vertical mobility, it reduces stamina, it could kill you in a small pond, and its usually noisy. An unarmoured character will generally be able to outrun an armoured one, all things being equal, and potentially outlast them. Also helmets generally make you look less cool, so there's that.