He doesn't normally have 5 guards on him it was just because
1) He became very violent
and 2) because Flint White was their, the warden of the place 
They have mostly dispersed now accept the 2 leading him away.