I think it's safe to say that anyone who's purpose is to trample the rights of others is in the wrong, because of your argument.
[quote=@Nytem4re]
the point of rights are to protect everyone, not just those you deem "worthy" of having rights. At that point it becomes a privilege and not a right at all.
[/quote]
Which makes Nazi idealogy inherently wrong. However...I agree with you. As long as their dreams/goals of trampling rights do not come to fruition, we have no right to take away their right of free speech, even if it's hateful. At least on a government level. If an individual is being affronted, we can't tell you what to do. But we can't make laws against it. 

[@Ace of Hearts]Also, people who follow the Nazi idealogy are in the wrong and I think we can all agree. But they are, essentially, people. Ignorant people who let hate control them, but people nonetheless. Talking to them and convincing them of being wrong should be paramount, rather than simply exterminating them or their rights. That would be a last resort.