To pick a concrete (if inflammatory) example of the subjectivity of "minority", perhaps it is worth considering the example of "Trump voters". Within the context of the US, they are less than 50% of the population, so would it be hate speech to insult his supporters?
To pick a smaller minority, what about "Baby Boomers". Should the "OK, boomer" meme be banned as hate speech? At 22% of the US population, they constitute a smaller minority than the proportion of non-white Americans (28%, excluding White Hispanics).