Nobody is arguing to make things harder or worse for those with privilege. Generally people are asking for equal consideration and opportunity and a recognition of past and existing injustice.
One argument against this is based on the philosophical concept of a Veil of Ignorance[1]. We should create the rules of society as if we don't already know what our position in society will be. No one wants to live in a society where they could be randomly selected for organ harvesting, for example.
As a result, it sounds like you're saying "screw you, I got mine". We can flip the question around: why does this majority deserve better opportunities? There aren't very good answers to that question.
The main thing I'd point out is that unlike your metaphor society is not a zero-sum game.
That's a massive leap in logic. Society isn't a simple one in, one out metric.
That is an argument no one is making. What are you talking about?
