As you said, we may very well be a couple iterations away from a chatbot that is truly indistinguishable from a human, but I still strongly assert that even a perfectly coherent chatbot is nothing more than an automaton and we humans are not automatons.
The fact that a couple replies in this thread made me feel defensive and a bit discouraged with their condescending tone is to me an internal reaction that an LLM or similar system will never have. Maybe an appropriate emotional reaction can be calculated and simulated, but I think the nature of the experience itself is truly beyond our current comprehension.
Maybe I’m grasping at the metaphysical to rationalize my fear that we’re on the cusp of understanding consciousness… and it turns out to be pretty boring and will be included with Microsoft O365 in a couple years.