"Even if we are to ignore all of this and pretend all was equal except for the oversimplified popular-history causes, it wouldn't be a proper comparison. It's callous to pretend that retaining slavery is an equal crime to not paying your taxes"

And its wrong to ignore the aspects of the movements that essentially maintained the same thing; Control over ones own affairs, and that means all affairs. Which includes slaves. And considering the best part of the CSA's economy was built off slavery, honestly I can see why they were reluctant to give it up, a crime or not.
Not to mention it was a cross road of history where morality was shifting... So it wasn't exactly a crime at the time. Playing devils advocate, although many European nations had abolished slavery years before, Russia didn't abolish serfdom until the 60's, middle eastern countries still retained slavery at the time etc. Its rather unfair to attach a crime to the CSA and their flag when morality of the time had only recently, in historical terms, shifted away
The result of the forced abolishment of slavery has given the Confederate flag a positive image of defiance in the eyes of southerners, it seems. People are going to have to deal with the cultural heritage of the civil war, even if they don't particularly like the flag. Banning it is a very dangerous precedent to set and honestly is a rather silly reaction to the confederate flag having been used by a few absolute crazies. The IRA used the Irish Tricolour, shall we ban that? Headchopping revolutionaries' used the French Tricolour, shall we ban that? The crucifix has been used by insane killers and racists too, shall we ban all protestant christians from using the icon of their faith? Of course not, these all sound like silly things to do. As an outsider looking in, I see people looking at a flag and saying the same thing; "The people who used this flag such and such years ago supported and did such and such, thus it should be banned because it can have a meaning associated with such and such, and thus crazy people who support such and such rally around it!" 

The problem is, if America does ban it as has been suggested, crazy racists will just find another symbol to use while the people who weren't using the flag for racist reasons will have been affected.