Let's take a scientific perspective of performance, ie using evidence to decide which hypotheses to believe.
If we have a situation that's repeatable, say in professional sports, we have a pretty useful basis for deciding who is better. For example a pro soccer player has his entire career recorded by the second. Every touch, tackle, sprint, shot, and so forth is recorded. On top of that the conditions are similar across matches: the point is to win. This is highly repeatable, noise can be removed by having many experiments.
What does a CEO do? Well they mostly do things that aren't replicable. If your company launches the iPhone, they aren't going to do it more than once. If they miss the boat, catching up is a different condition to leading. If the economic cycles turn, the CEO's time at the helm will certainly not last enough cycles to tell whether they are good at weathering such storms. In fact decisions that CEOs make tend to be rare and unrepeatable. Did Marissa Mayer have to be skilled to sell Yahoo? Well nobody else has been allowed to try, so we don't have a lot to compare with. Did she get a good price? Well she only did it once, so we don't know.
In fact it is a lot easier to ascertain that LeBron James and Cristiano Ronaldo are at the tops of their games than any CEO. Do they score a lot? Yes, and many other dudes have tried. Are they effective compared to opportunities? Yes, and the stats say so.
Coming back to the CEO vs man on the street, I get the feeling a lot of CEOs are quite replaceable. Their firm is in some business, and there's a few strategic decisions to make. If you have multiple parallel universes, it's possible the CEOs would make the best choice more than the man on the street. But you'd certainly find men on the street who'd also launch the iPhone if given the chance. Now consider people who are reasonably close to senior management but not yet there. Would they make similar decisions? They certainly cost less.
By contrast it is glaringly obvious when a man on the street tries to play a sport. It's even obvious when a pro who is less good tries to play against a superstar.
