The Confederate flag means different things for different people. For most minorities, it means more to us than, "the owner of this truck or house is from a rural area", it means to us that this person probably hates us, that we should try to avoid him. It makes people feel extremely unsafe. 

Flying the Confedarate flag privately is fine, so that guy with his truck decked out with the rebel flag really has nothing to worry about. But imagine, if you're a citizen of some state, and that same flag that makes you so alienated, uncomfortable, and unwanted on municipal buildings, police stations, schools, houses of legislature, can you see the problem?