I won't say it can't be done but you'd have to be really lucky to drop everyone with a single punch, the skull is surprisingly resilient. Then there is the range issue because it was pointed out in the manga that if they get a grip on you they can't be shaken off easily and punching could put others close enough to grab you but the same can be said of using a bat or sword, the only advantage to these weapons is a slight increase in range.