E.g. the training data might look like "a fooble is to a tooble as a fabble is to a" with the answer "tabble".

So you feed it tons of these kind of nonsense training data that forces it to only learn the in-context reasoning part of language, and none of the world knowledge.

That said, it is unclear to me how much value such a model would be.

You could imagine a middle ground where it does have basic knowledge, such as dictionary definitions and things derivable from those.

E.g. "A tree has leaves. Leaves are green. Therefore a tree has a part that is green." type stuff.

So you could give it some amount of world-grounding and common sense knowledge, but nothing involving history, proper nouns, etc.

I imagine you could make such a model much smaller than these giant LLMs.