Those that actually used them in the 20th century (like they did in Asia, not some ancient vikings or whatever) still use it.
And that symbol was 100% associated with the Nazis in the West in the 20th century. Nobody used it at the time before the war for anything else, except some tiny fringe.
If it was some mainstream symbol or idiom, merely co-adopted, we'd probably still be using it too.
If the Nazis used the cross for example,people wouldn't stop using the sign of the cross.