Not always. There's the illusion caster who can make him see something that's not and take him down without him aware. There's the the blue haired lady, with a few well placed and paced hits she can break down the field.

He's not invincible, you just have to strategize a little bit