Here’s another take on the Dunning–Kruger effect. I made two main points:
1. Consider N independent observations of two variables, X1 and X2, with imperfect correlation. Next, we assign percentiles to them: P1 and P2, respectively. We then take a subsample of size N/4 with the lowest values of P1. The statistical fact, not psychological effect, is that, in this subsample, the average of P1 will always be less than the average of P2. With a large enough sample size, the difference will be statistically significant.
2. Percentile is a measure with bounded support (between 0 and 100). It's not correct to use it as a measure of abilities in this experiment. Participants from the top test score quartile can overestimate their abilities by a maximum of 24 percentiles and an average of 12 percentiles. Participants from the bottom test score quartile can overestimate their abilities by a maximum of 99 percentiles and an average of 87 percentiles. There is certainly a bias here.
The first point is not particularly new; there are published papers on this topic. However, not everyone is aware of this, so it's worth mentioning. I also provided a Streamlit app for simulations and a source code.
The second point is somewhat novel. At least, I haven't encountered any references to it in the context of the Dunning–Kruger experiment.
I also encourage you to think about the question: if a person scores the maximum number of points in a test, does this mean that they cannot overestimate their abilities?
