Yes, we're not going to get to the bottom of it here. Since most of us aren't social scientists and don't have experience with any particular diagnostic tool, all we are doing is comparing priors, and there's no reason why you should take the priors of some other stranger commenting on the Internet all that seriously.

But I'll say that my priors lean heavily towards the idea that most surveys, even the standard ones, are less meaningful than they appear, for reasons explained here:

https://carcinisation.com/2020/12/11/survey-chicken/

> which is perfectly acceptable when done right

Yes, perfectly acceptable. But garbage for the reasons I described above. You can probably tell that I don’t have a high opinion of social science studies, so it’s unlikely that I’ll be convinced that this is one of the vanishingly small number of studies that I’ll concede has any validity at all.

It’s garbage because it tries to assign a quantity to an English word that already means something. They are piggybacking the formal (which is limited in scope and meaning) on the informal (which is deep and reflective of human experience). If Jane Goodall makes an observation about aggressiveness in social situations it’s a judgement that I can easily accept. There’s great value in trying to obtain a detailed qualitative understanding of behavior.

On the other hand it’s one of the great mistakes of the social sciences to assume that quantification is necessary for engaging in science; this results in “metricism” or “scientist”, which cargo cults the trappings of the physical sciences without any of their insightful power.

Of course the conclusion of the article is correct, but I would not consider this evidence for that conclusion. If they had reached the opposite conclusion I would also have dismissed it entirely, so it’s only fair I do the same even though it lines up with my personal beliefs and experiences.