undefined | Better HN

0 pointssmcin1y ago0 comments

If you google "Which Kahneman claims were wrong", here are the first two of 4.97m hits"

https://slate.com/technology/2016/12/kahneman-and-tversky-re...

https://replicationindex.com/category/thinking-fast-and-slow...

> "Table 1 shows the number of results that were available and the R-Index for chapters that mentioned empirical results. The chapters vary dramatically in terms of the number of studies that are presented (Table 1). The number of results ranges from 2 for chapters 14 and 16 to 55 for Chapter 5. For small sets of studies, the R-Index may not be very reliable, but it is all we have unless we do a careful analysis of each effect and replication studies.

> Chapter 4 is the priming chapter that we carefully analyzed (Schimmack, Heene, & Kesavan, 2017). Table 1 shows that Chapter 4 is the worst chapter with an R-Index of 19. An R-Index below 50 implies that there is a less than 50% chance that a result will replicate. Tversky and Kahneman (1971) themselves warned against studies that provide so little evidence for a hypothesis. A 50% probability of answering multiple choice questions correctly is also used to fail students. So, we decided to give chapters with an R-Index below 50 a failing grade. Other chapters with failing grades are Chapter 3, 6, 7, 11, 14, 16. Chapter 24 has the highest highest score (80, which is an A- in the Canadian grading scheme), but there are only 8 results.

0 comments

faeriechangling1y ago

Which is to say in other words, most of the book probably replicates, particularly the parts based on Kahneman’s own work, and for the parts that don’t you can just skip the chapters or take them with a grain of salt.

Kahneman to me always struck me as the one eyed king of the replication crisis. Yes he fucked up but he fucked up notably less than his contemporaries and most of his work is still readable.

smcinOP1y ago

Without a copy of the book, I don't remember which parts were based on Kahneman’s own work, and I don't see that we can/should just skip the other chapters.

R-index guys said [0]: "Table 1 shows the number of results that were available and the R-Index for chapters that mentioned empirical results."

Chapters where estimated R-index < 50: Ch 3,4,6,7,11,14,16

Chapters where estimated R-index > 50: Ch 5,8,9,12,17,24

Chapters that don't cite empirical results (by Kahneman, or who?): 1,2,10,13,15,18-23, all of 25-38

As to the chapters that had empirical results, and had an estimated R-index > 50: scores of 55, 57, 60, 62 are really scraping by; saying that means they "probably replicate" is setting the bar really low, even quoting Tversky and Kahneman (1971) back at themselves. (The R-index guys say "Even some of the studies with a high R-Index seem questionable with the hindsight of 2020.")

As to whether he was the one-eyed king of the replication crisis, he certainly started speaking out in 2012 [3] after the social priming scandal broke; did insiders have suspicions about non-replicability before that and should people have pushed back more, earlier? The fallout from the Francesca Gino and Ariely scandals continues.

[0]: https://i0.wp.com/replicationindex.com/wp-content/uploads/20...

[1]: Chapter listing: https://catalog.lib.uchicago.edu/vufind/Record/8467452/TOC

[2]: Chapter summaries by Conor Dewey: https://www.conordewey.com/blog/every-chapter-of-thinking-fa...

[3]: Nature (2012) https://www.nature.com/articles/nature.2012.11535

resource_waste1y ago

>Which is to say in other words, most of the book probably replicates

This is extremely irresponsible.

j / k navigate · click thread line to collapse