I can imagine some people I know claiming I'm not Hufflepuff for being 'cruel, cold and/or uncaring'.
But those are cases where people confuse honesty with hatred. Most people prefer to sugar coat what they say, while I respect the person enough to tell something to them straight.

In other words, if you see Hufflepuff as more of the "Bubbly, Grinny, Poppies and Roses. Let's do everything with a smile", I'm not a hufflepuff by a long shot. But if you see them more as "Kind, caring, wants everyone to get along", I'd say that I fall under that house.