Nobody actually makes this argument though.
By the way, I know it's a parody of another story that makes this exact refutation. But I think this only serves to highlight the point.
It reminds me, oddly, of the debate over whether video games can be "art". A turning point was when they actually did something that art does: [evoke profound emotion and thoughtfulness](https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy) for the player.
(And before that, "[Can photography be art](https://daily.jstor.org/when-photography-was-not-art/)?")
We may not come to something as simple as "machines can be conscious", but we will certainly have to understand consciousness better if we want to refine our questions.
---
Edit: My point is that we don't need to be angry, but we may have to tolerate people expressing their exploration through overly-confident language, and be patient with that.
And Ted here is obviously exploring. His examination of Claude's constitution clearly shows some nuance. He asks:
> So, given that Claude is not conscious, what are we to make of Claude’s constitution?
And his conclusions are split, between this is useful and this is dishonest. It's a great tension IMO.
> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
I also remember the "video games are art" debate and the fury from one side of the aisle. I agree that a better understanding of the opposite side should have been part of the debate. But I don't believe that debate was existential. A better comparison to me is the climate change debate. I'm fine with having that debate in an environment where there is little at stake. But it's too late to be doing it with policymakers; we need to be talking about what to do.
https://www.goodreads.com/en/book/show/217432753-the-ai-con
which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper:
https://zenodo.org/records/20071869
which says,
> [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.
And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.
