I don't think I've ever seen anything about physical violence being associated with Muggles. Physical violence is frowned on in the wizarding world for the same reasons as in ours. Don't forget George and Harry's fight after the Quidditch match. In fact quidditch itself involves a lot of physical violence. Malfoy himself keeps Crabbe and Goyle around, and there's no way they're there for anything other than muscle. Wizards or not, people are gonna hit other people when they get mad.