Not everyone deserves a voice, but even if you do we don't have to listen. We don't have to tolerate intolerance either, we are perfectly free to not tolerate it, in fact we must.
If you are curious who decides about who deserves a voice and who don't this is something they teach in high school, though not everyone receives civic education unfortunately.