If you have only "the rights we want" then your rights are subject to the whims of the majority.

In America our tradition of civil liberties protected the Illinois Nazis' freedom of speech as they marched through Skokie, celebrating the enemies of America who, within living memory, had waged war and killed millions of our citizens. The exercise meant as an affront to our nation and everything it values. We celebrated it, because if our worst enemies can have rights we don't like, then everyone can have rights.

In the UK, you can't even tolerate Dankula, and people think it's wonderful and good that he was fined for the nazi pug incident, and no one has really has rights unless they're popular. You'd think incidents like that would wave a few red flags, given that your nations' checkered moral history includes multi-century exercises in the deliberate oppression of your religious enemies.