I’ve been writing lots of Prolog recently and asking ChatGPT questions. Many of my questions have been sincere but a bit like the bear trainer — ridiculous for someone who knows what they are doing. Meanwhile ChatGPT will answer it as if the premise is valid. The answer is valid sounding nonsense, which may lead you on a wild goose chase — a bit like the aspiring space bear trainer.

It isn’t assuming that I’m asking from a “hypothetical and for a potentially fictional purpose”. If GPT is in effect a conditional probability distribution over tokens, it isn’t “assuming” at all.

This IMO is a clear challenge to sense making for ChatGPT which is not obviously fixable through fine tuning. I don’t think factfulness is either because low contrast examples are hard to train for, especially if they are compounds of true things . Eg “tell me about logic regression”