Racists in positions of power may tilt the playing field in your favor whether you've asked for it or not. And it may not even be you personally, but people in your neighborhood, your social groups in general.
Unsafe generalization is at the root of prejudice. Racists / sexists / bigots generalize from the worst instances of individual behaviour to a group, or aggregate statistics about a group, and then apply the generalization in specific individual scenarios. A crude example, taking the generalization "Jews run global finance" - and it's true that they have been historically over-represented - and then applying the generalization to specifics: "you're a Jew, I don't like Jews because they run the world".
A good rule to bear in mind before leaping to prejudice is that the variance within groups is larger than the variance between groups.
White privilege is a prejudice concept built along the same architecture as racism and sexism. It takes aggregate group attributes and tries to enforce it in the particular against individuals. You've just rehearsed the line yourself - "you benefit from racism" - you've given an example of instantiation of a group attribute upon an individual without evidence. It is literally prejudice, and it's unjust, even if it's more likely to be true than false.
I have zero doubt that many people are treated differently based on skin and other overt characteristics. But that's not my argument.
The core of the issue is that this is a stereotype being applied on a person because of their race. At no point is someone going through that person's own individual history and determining specific points where they have benefited. Instead, there is just the stereotype (which might even be true on average) that is being applied to each individual person because of their (perceived) race.
What was the word for stereotyping someone based on their race?