Pretty much.

> You're just disagreeing with my wording.

Yeah, that's the "going even further" part. Words are important. The importance of words is really foundational for a lot of what we are talking about.

> What you're describing is, it doesn't think, it doesn't understand, it just follows whatever patterns in its training corpus kind of correlate with patterns in the prompt. And I agree.

Cool. It's nice to be on the same page. Let's read the next one:

> Where I may differ is how to regard the training corpus. True, it contains a massive amount of distilled thought, encoded in words.

That's the biggest point I'm trying to make by "disagreeing with your wording". Words have significant implications. In this case, the implication is "problem domain": language vs. text. Language is a subdomain of written text. Written text is a subdomain of all possible text. GPT is dealing with written text.

GPT does not only model the language patterns that humans intentionally put into text. GPT models all of the patterns that exist in the text.

Humans, when writing, do "distill" thought into language; but the result is messy: we end up encoding more patterns than we intended to. Our behavior, our choices, our reasons, and the surrounding situation we exist in, all inform our writing. Those are the patterns that get encoded into the text, right alongside language.

GPT doesn't see any pattern as different from another; so it's misleading to say that it models "language". GPT doesn't categorize that specific set of patterns: it models all patterns equally.

We humans have a good idea of what to expect from only language patterns. If that were all that was modeled, GPT's output wouldn't be nearly as interesting to us.

GPT's output is interesting to us. If we look objectively at the model it generates, we can see why: there are patterns there we had no intention of writing; and GPT modeled them just like it modeled the language-specific patterns we expected it to.