The paradox of tolerance was aimed at those who would with "fists or pistols" prevent others from sharing their views and was premised on the right of self-defense.
In historical context, it's seems squarely aimed at the paramilitary organizations of various movements popular around the 1930s or so who physically injured people for saying things they disliked.
