The key thing you are forgetting when you say 'shut up, straight white boy'[1] is that throughout most of history (and still today), even if you were a white male, you were statistically speaking poor and fairly unimportant, and also shut out of 'the conversation'.
There is a fairly significant difference between accepting additional input from those with a different perspective (inclusive) and excluding input because of perceived privilege based on race/gender/etc (exclusive).
[1] I'm adapting this from the title of a feminist blog post critical of the term 'privilege' as a useful rhetorical device, which curiously cannot be found (404), don't take it literally...