In respect, I can understand your concern. Sir James Gilford is more or less a fanatic rather than a rank and file soldier, although he is head of the lesser equipped and even less motivated Hylain Town Watch. Due to his attitude, the stress he puts on himself that makes him unlikable, his men would only be of concern if Maryov were to strike him down. 

Plus, it hasn't been that long since a 'particular Chilfos' has gone missing and that caused quite a commotion, so he might be a bit uppity if he suspects a brawl or clashing of swords.

Edit: If that still troubles you, I can work on another post that takes advantage of his posse of drunk goons that are always telling him to chill.