The actual counterpoint is demonstrated in _Blindsight_ Peter Watts. He makes a strong (and rather terrifyingly strong) point that intelligence is not consciousness.
My original comment (roughly "there's no intelligence in this article, nor sentience in LLMs") is in response to the blog post's buried lede (that the cumulative activity of LLMs has accrued to a weight of "AGI is around the corner" or "there is artificial consciousness in this matrix").
To be clear, I'm not saying LLMs are useless or a wrong direction in development of "AI," but rather it's the Fool's Gold for the path towards AGI, the pursuit of the academic field of Artificial Intelligence research. A research that I've been abreast of for years before this new age of language models that has made everyone with a keyboard an arm chair expert.
Also, thank you for the book recommendation, it's on my list! :)
So I still disagree with your elucidated point (as you end with "which is valid"): the OP author is using prior art fiction to bolster their opinion of LLM-based software tools as being a possible vector of sentience, not to disarm our chauvinism like the original author intended. If OP wanted to make that point, they could have written a critical essay instead of farming out their thoughts as tokens.
But still, I look forward to reading the book you suggested to understand and appreciate your perspective more.
I think I mostly latched onto the ship being called Rorschach and the humans immediately proceeding to torture its inhabitants. That felt very relevant to me and sort of overshadowed the actual consciousness question.
Also I don't see why intelligence not being consciousness is scary? My cats are very conscious as far as I can tell, but not particularly intelligent. I think LLM's exhibit some contextual intelligence without there being any particular reason to believe they're conscious other than woo psuedoscience.
That said, I don’t think it is useful for philosophy nor science to consider intelligence to be the same thing as consciousness. In fact I would go even further and claim that intelligence is not a useful construct, neither for philosophy nor for science. Consciousness, on the other hand, I think is useful for philosophy, but not (as of now) for science.
