You can ignore anything said in chapter 4 about priming for example.
See https://replicationindex.com/2020/12/30/a-meta-scientific-pe... for more.
This, IMO, is the actual biggest problem with LLMs training on whatever the biggest text corpus us that's available: they don't account for the fact that not all text is equally worthy of next-token-predicting. This problem is completely solvable, almost trivially so, but I haven't seen anyone publicly describe a (scaled, in production) solution yet.
It's probably the AI thing I'm most excited about, and I suspect we're not far from that, although I'm betting the copyright battles are the primary obstacle to such a future at this point.
It's not just that individual studies have failed to replicate. The whole field was in question at least a decade before the book was written, and since then many of the foundational results have failed to replicate. The book was in a sense the last hurrah of a theory that was on its way out, and the replication crisis administered the coup de grace IMO.
I want something similar but for children's literature. From Ralph and the Motorcycle to Peter Pan, a lot of stuff doesn't hold up.
The books provide plenty of utility. But many things don't hold up to modern thinking. LLMs provide the opportunity to embrace classic content while throwing off the need to improvise as one parses outmoded ideas.
You could not redact piece of art out of "old ideas". It is like re-drawing classics paintings but mask nipples and removing blood.
And books which could be redacted this way without falling apart — well, don't read such books at all and don't feed them to the children.
Literature for children must not be dumbed down, but exactly as for adults, but better.
Also please omit "who has a better story than bran"
It's saying that the author's invented metric may indicate that the studies within each chapter may not replicate. No actual replication studies were done to produce the table in that post.
I included that blog post as a guide to what other results may be suspect.
Long before the controversy was public I was reading that book and, despite claims that the reader must believe the findings, it sounded like nonsense to me. So I looked up the original paper to see what the experiment set up was, and it was unquestionably a ridiculous conclusion to draw from a deeply flawed experiment.
I still never understood how that chapter got through without anyone else having the same reaction I did.
As he's shifted from primarily a journalist to primarily a storyteller, he's chosen to sacrifice additional information and accuracy in lieu of telling a consistent and compelling narrative that support what he thinks is the important thing to take away, not necessarily what you would take away were you to review all the same information yourself.
Under that understanding, I find him fun to listen to. The things he "reports" on/illuminates are interesting, but at this point I don't assume he's giving them an even handed representation, so his conclusions are not necessarily my own, and at best it's a set of things to look into and research myself if I find my interest piqued after a fun or interesting story is told.
And that could be for a number of reasons. Of course, sometimes the results are just wrong, due to statistical flukes, or too creative data cleaning and analysis. Often the results might just be much more limited than what the original study claims: Maybe the results of a psychological study is valid for MIT students in the beginning of the semester, before lunch, but not for Yale students in the early afternoon. In this case the only mistake would have been to assume the results were universal.
This tendency seems to be capitalized on fairly heavily in political media by finding some poorly supported assertion of the other side to criticize, which causes people to assume the opposite is true.
But such a small fraction of possible propositions are true that it is unlikely to be worthwhile to waste much time on propositions with no evidence.
I'll have you know you just nearly nerd sniped a mathematician ;-)
I suspect I may not be the first person to entertain this question, perhaps there is some literature on the matter.
