While being a free speech advocate is political, it does not fit into the US left/right political structure. So I don't see opposing the censorship of political speech as being "the other side's politics."
It's funny because I clearly remember when it was the other side with a "moral majority" who was trying to silence all opposing views. I was against it then as well. I think we should all be against authoritarianism no matter which political party we support. As I see it authoritarianism has led to suffering of the common people almost every time.