> Essentially you should be tolerant of others, unless they themselves are intolerant.
No, it's called a paradox because there is no "easy" answer to it. Also, who determines the exact line between free speech and intolerance? It is impossible to do. The reason the world is polarised today is because everybody thinks they have the simple answer to a literal ethical paradox that has none. The simple answer being: "those that share my beliefs are OK, those that don't should be canceled/demonised/hidden somewhere I don't see them." Thus creating echo chambers so people get more and more radical.
Free speech is the only half acceptable answer, which means free speech for anyone, whatever their point of view, however offensive you deem them to be. Rowan Atkinson explained it more eloquently that I ever could: https://www.youtube.com/watch?v=xUezfuy8Qpc