Your POV is simply, factually, wrong; more people get high SATs and apply to the Ivy League schools than they can admit by a wide margin, and there are a lot of tricks (many of which leverage accidents of birth, wealth, and social circle) beyond that which go to getting to the top of the list.

The bigger ones involve the hyper rich buying buildings, and the merely very rich engaging in bribery which is higher risk when discovered, but legacies, padding applications with extracurrriculars for which opportunities are very much not equal, etc., all play a role.