"By the same logic how can we allow Humans to do those jobs either?"

First, since you can't see tone, let me acknowledge this is a fair question, and this answer is in the spirit of exploration and not "you should have known this" or anything like that.

The answer is a spin on what I said in my first post. Human failures have a shape to them. You cite an example that is certainly common, and you and I know what it means. Or at least, what it probabilistically means. It is unfortunate if someone with lesser understanding calls in and gets that answer, but at least they can learn.

If there were a perfect support system, that would be preferable, but for now, this is as good as it gets.

A computer system will spin a much wider variety of confabulated garbage, and it is much harder to tell the difference between GPT text that is correct, GPT text that is almost correct but contains subtle errors, and GPT text that sounds very convincing but is totally wrong. The problem isn't that humans are always right and computers are always wrong; the problem is that the bar for being able to tell if the answer is correct is quite significantly raised for me as someone calling in for GPT-based technologies.

I don't think you're right. Even if you add "correct" context, and in many of these cases "I can locate correct context" already means the GPT-tech isn't adding much, GPT still as absolutely no guard rails stopping it from confabulating. It might confabulate something else, but it still confabulating.

Fundamentally, GPT is a technology for building convincing confabulations, and we hope that if we keep pounding on it and making it bigger we can get those confabulations to converge on reality. I do not mean this as an insult, I mean it as a reasonable description of the underlying technology. This is, fundamentally, not a sane way to build most of the systems I see people trying to build with it. AI Dungeon is a good use because the whole point of AI Dungeon is to confabulate at scale. This works with the strengths of GPT-like tech (technically, "transformer-based tech" is probably a closer term but nobody knows what that is).

This hangs on what it means to "take it into consideration." If you gave me new information, I would attempt to see it in context, evaluate its relevance, and either update my positions accordingly or explain why I do not see it making a difference. If I saw difficulties doing this, I would ask for clarification, explaining what it was that was seemed difficult or unclear.

As far as I can tell, there is no reason to think that the way GPT-3 generates its responses could possibly result in this happening - even the basic ability of correctly inferring corollaries from a collection of facts seems beyond what those methods could deliver, except insofar as the syntax of their expression matches common patterns in the corpus of human language use. And the empirical results so far, while being impressive and thought-provoking in many ways, support this skepticism.

Did we need digital painting tools? Paint and Easel worked just fine. Did we need paint and easels? Cave walls and clay pigments worked just fine. Do we need Automobiles or Trains? Horses worked just fine. Etc. Etc. Etc.