A metric of diversity being applied to selection really doesn't mean it's the sole criterion. The effect isn't negligible, but neither is it fully-like wannabe egalitarian marxist systems where your parent's job or political affiliations forever determined your future status without negotiation.
There's definitely a scale between a meritocratic and egalitarian society.