I'm being snarky there, but I've genuinely never seen people make the arguments you're talking about in real life, and I run in some fairly lefty circles. Maybe online, but even then I rarely see people actually trying to argue that privilege has anything to do with the validity of people's opinions. More common is the idea that we need to better promote the voices of those with less privilege, which I don't see as being a particularly objectionable idea.

The only place I regularly see the points you mention are in the opinion pieces of certain types of pundits who like to peddle outrage and invent menaces that don't exist. They regularly tell me that people say those sorts of things, but rarely seem to be able to provide receipts.