To be clear, there are racists, and there are valid accusations of racism. I grew up and lived in the American Southeast for a long time. I'm WELL aware of the ongoing existence of and trouble with racists.
While my personal experience is affected by being in the SF Bay Area for the past couple of decades, it is my experience that over those decades the overwhelming majority of accusations of racism in ordinary conversation are not invoked because of racism, but are used as a shorthand for "Fuck you, I'm done with this conversation and done with YOU.".