So, you say, "Write me a heap for Java", and it spits out amazing, wonderful code that is almost right. Well, yeah, that's cool and all, though I would point out that its training data probably included a heap that was entirely correct, but, still, it's a common question.

But I'm not writing a heap for Java. I'm taking business object X and stuffing through system Y and feeding the result to system Z, all internal to the company I work for. Good luck with getting ChatGPT to do that.

But it deceives people, because the very questions they are most likely to ask as a test are the exact questions it knows the answer to, for the very reason that they are also the most likely questions to be in the training data.

(This article kind of doubles as great set of examples of just how quickly you get into confident nonsense if you stick a toe slightly out of line in your query. For instance, even if it knows "What is Y" and "What is Z", it can't necessarily combine them in any correct way. It'll spew out a lot of words that sound very confident, but it'll make mistakes without any indication they were mistakes.)