This is just misguided, plain and simple. I am white, male, heterosexual, and from a middle class family, and I absolutely know that my background gives me privilege. It doesn't matter if I earned it or not; I had and still have an advantage over other classes of people, and that is where the hostility is directed.
Instead of taking it personally, use this advantage to eliminate as much privilege as possible. Don't shy away from "leadership," but instead embrace it and make diversity a major focus of your leadership style.
Calling out the underrepresented as "bigots" for being upset is just plain cowardice.