The funny thing about ChatGPT is it will write code that uses non-existent confabulated APIs. You have to then call it out and it will say, oh sorry, of course you're right, here's another confabulated API, etc. The amount of convincing B.S it can spew is enormous!

Worse, when you stray into topics that are controversial it will often use informal fallacies. When you call it out, it will say, yes, you're right, I used an informal fallacy, here is what I should have said about controversial topic that's not the party line, and because you're so smart I won't b.s you.

Running with the analogy.. We’re used to using tools that help us find needles in the haystack. A rake, a bright light, a metal detector. Now someone sold us a machine that turns hay into needles. They’re not quite as good, but they’re definitely needles. So now as you say the haystack is going to get covered in these. Do we ban use of this machine? Build new machines to separate these synthetic needles from real ones? Or improve the machine so that the needles it makes are good enough for what we need?