But this is a feeling that must be let go. Privilege allows people to reach excellence and excellence is scarce, so no, privileged people that do their homework won't suffer because we are trying to do the right thing, allowed by our current stage of civilization that generates so much surplus.
Not picking up the capable people and letting them reach their level of excellence is a big problem in our society, and everybody would be better off if this was fixed.