Sure, the swastika has a history predating the Nazi party, but the association that people who use it in a modified SP logo are not comparing Smart Parts to Hindu or Native American symbols- the implication is that Smart Parts is equivalent to the Nazis. That is ridiculous hyperbole. As far as I know the Gardners have no official position on eugenics and no program of genocide, and they certainly aren't socialists.
I believe this is one of those issues that is more sensitive in Europe than the US. When I was in Germany, a kid on a train made a Nazi joke, which was in poor taste maybe, but the other kids reacted in a much more offended manner than I would have anticipated. I think that association with Nazis is almost always an epithet, but in Europe there seems to be a lot more shock attached to it.
Also, using Hitler or Nazis in an arguement damages credibility. If you are willing to exagerate enough to bust the Hitler card, you are trying to manipulate the feelings of the other party (Hitler = evil), rather than persuade the other party with reason.