It turns out if you had a super power where you always knew how to pick the next best word for a sentence, you could convince a lot of people you know stuff.
Is knowledge nothing more than a series of words in correct order?
In political discussions, a lot of humans don’t really know what the fuck they’re talking about, but they do know what to say in response to certain stimulus. They know the talking points, the key phrases, the terms, they parrot these back to you when you provoke them to say it. They are basically human sentence predictors. Stuff comes out of their mouth based on what they’ve been trained to think the next word should be. They don’t pause to reflect on some gathered knowledge and then present an observation to the world.
This is basically what GPT is, but with everything. And the only way GPT plays a game, is if the state of the game can sufficiently activate some output that represents the next best move for the game being played, which isn’t really a readily available data set for all games, especially for made up games.
We've all been "trained" with various facts and when two people meet who have been trained on substantially different bodies of knowledge/facts/experiences it can be very difficult to find common ground.
FWIW, I asked ChatGPT to give me a short list of cognitive biases and psychological phenomena. Interesting to think how many of these are dependent on our personal "training data":
Confirmation Bias
Cognitive Dissonance
Anchoring Bias
Belief Perseverance
Groupthink
Ingroup Bias
Sunk Cost Fallacy
Motivated ReasoningBut the question remains why this is possible! A good enough pure predictor could play novel games or devise new theorems, but GPT absolutely can't. It can however give confident explanations, as well as write passable code and occasionally even do some novel problem solving (simple, impressive only because it comes from a computer, but still there). The question of why it can do some things and not others is interesting, and can't be swept under the rug just by reiterating that it's a predictor.
Does the structure of language really do such a good job of conveying information that GPT can operate on it blindly and get results, Blindsight style? Is composing prose far easier than we expect, leaving the bulk of the model free to do a tiny amount of "reasoning" that we find unjustly impressive because of how well it's presented? Is it handicapped primarily by the fact that it can only carry out extremely short computations, and can't be trained to use chain-of-reasoning to get around the limitation? We have no idea what, if any, inherent limitations predictive models have. We have no idea why GPT-sized models are good at the things they are, and bad at the things they aren't.
(We don't know)
Not sure what else I can really say about that. They are not the same.