i'm of the opinion that it's like self driving cars. Even if you get 99.999% of the way there it's still "not anywhere close" to the real thing because you're speaking with something that has little to no agency and acting as though it's a good substitute for a person.
My instincts tell me that humans are pretty good at detecting this difference. And when they aren't - they still won't like being lied to or tricked about it. You can see it already - generative art, or music for instance is (in some cases) objectively more impressive than art created by humans all else constant. You might trick a contest into giving you an award but the moment people find out it's generated, they almost immediately react angrily and no longer express interest in the result.
That's because they used to attribute the result to a person and now they know it's not a person. The psychology there probably isn't even fully fleshed out, but i feel it instinctively, as I said before. And I suspect others do as well based on the reactions here.
Sorry for assuming bad faith. i've met a lot of persons here who really do think LLM's in their current form are a kind of sentience. Blake Lemoigne (sp?) is a good example of that kind of naïveté.
I too have a human therapist, doctors, etc. And I too find myself chatting with ChatGPT, etc. about personal issues and in certain cases benefit tremendously from it. In particular, whenever it is something I would normally feel embarassed to say to another actual human. Since I am very confident ChatGPT doesn't have feelings or even an internal monologue with which it could "judge" me - I have no issue telling it such things. The benefit here is from the questions I can have answered that would otherwise go unanswered. I think this makes for a potential assistive technology as you implied earlier (better than a worksheet).
But for precisely that same reason, it will never work (in its current form) as a complete substitute for a human. And attempts to do so may in fact be actively harmful (as I originally suggested). Again, I'll just say that I don't think there's yet enough research on this but that "I know it when I see it". Any sufficiently serious topic I discuss with ChatGPT ultimately winds up with me drained because I feel as though I'm talking to a wall and not actually being acknowledged by anyone with agency who matters to me.
I will definitely admit that this is a highly opinionated take and is rooted in a lot of my personal feelings on the matter. As such, I can't really say that I've definitively proven that my point is the correct point. But, I hope you at least get the gist of what I am saying.
