To put it as well as I can : after my 2nd reaction I went on to read the now available 2nd part, which also linked to this Open Letter by Yudkowski a month ago, which spells out my 2nd reaction probably better than I could :
https://news.ycombinator.com/item?id=35364833
> I agree that current AIs are probably just imitating talk of self-awareness from their training data. But I mark that, with how little insight we have into these systems’ internals, we do not actually know.
Also, I am not convinced that our current research won't hit a brick wall well before super-intelligence, which would then require a radically different approach... but why are we willing to gamble with something as dangerous as this ?!
Don't we? Maybe someone with more architectural knowledge of LLM systems can fill in the gaps in my knowledge, but where's the feedback loop that would allow actual self-awareness to develop? FAFAIK, LLM's are feed-forward networks. There are neither small cycles spanning a few nodes, nor integral cycles allowing the model to see what it's outputting -- the only feedback it gets is through the training process, which does not act as a mirror.
I struggle to see how self-awareness would ever manifest from a system that cannot experience its own output.
Sydney has no genitalia, no hormones, no sex drive. It does not "desire" someone. It may have a theory of mind, but clearly any AI saying that it loves someone is just working from its training data. The article treats this as though there was an actual intention there, rather than just a model repeating its training [0].
Just as obviously, this is a journo playing to our human desire to prevent aliens from running away with our women - any thought of non-humans breeding with humans triggers something pretty deep in us. Well played, sir. Take your paycheck, sit down, and please stop pushing the buttons.
The AIs are not going to take over the world in the sense that this article talks about. They are not invented by supernatural forces beyond our ken, as it suggests. This is a human emotional reaction to things that appear to be intelligent. I agree that this generation of AI is going to hit a brick wall and the shortcomings will become obvious, but the emotional reaction will be the same. I think it will be another 10-20 years before we solve some of the technical challenges in this generation of AI. But we're having the discussion about AI, and the emotional reaction to it, now. Which is probably good, because by the time we develop actual AI we'll have been living with our current level of "almost-AI" since we were kids and we'll be used to it.
And to answer the question: Every technology is dangerous. That has never stopped us from using it.
[0] Though, of course, that raises questions about what an "intention" is in human terms, too. Are we just "repeating our training"? I think not, at least for sex & love - there are too many hormones involved.
His part Two is available (to subscribers at least)
> I cheated a bit there, I admit. I changed one of the words. The name that the saint used in that passage was not ‘Ahriman’. It was ‘Antichrist.’
That's manipulation. That's gross. That's a shame. There are some interesting ideas in the post.
He's illustrating how a description of the Antichrist from over 100 years ago so neatly describes modern man's deity.
You clearly understood the parallel, for you felt manipulated. His manipulation is not only intentional, it is admitted by the author!
If it shook you up so much that you had to stop reading then it was quite successful.
I’m talking about our first AIs, the corporations. They maximize their goals at any cost, live forever, assimilate other corporations, and frequently trick humans to engage in unsafe behavior for their own benefit. They expand to control more and more territory, concentrate power to themselves, and seek to make humans dependent on them — while at the same time polluting everywhere to make the human race weaker. They don’t intend this, they just have an alien code of ethics that worships shareholder value as the height of virtue.
How is computer AI going to be worse than what we’re already doing?
I don’t think “AI = corporation” is a useful framework, current AI doesn’t particularly resemble corporations and corporations are much more human and less “intelligent” than I’d expect future AI to be.
I had read up until there, but stepped away to do something else. I was looking forward to coming back to something more ambitious beyond the recent spate of "AI" talk, like some kind of analysis within a larger context of the forces pushing us towards technological/economic determinism (slash digital authoritarianism) for the past few decades - the elephant in the room that all of these alarmist "OMGAI" articles conveniently leave out.
Instead, this article disappointingly nose dived into the superstitious/religious, with its characteristic lack of constructive ideas - essentially just lamenting that we're no longer in the past when things were oh-so-simple. Not that there isn't some fundamental wisdom in the religious tales and prognostications of old, but to take them literally as descriptive models that are directly applicable to novel phenomena is utterly vacuous.
This is starkly demonstrated in the second installment of the essay, where the author leans on "I won’t have a smartphone in the house" as if that is an example of ascetic restraint. Sorry, abstention is not principled engagement!
I've developed my own personal philosophy mostly revolving around libre software and personal computing autonomy, which I think has done a decent job of letting me dodge many attention-economy bullets. But it doesn't feel particularly apt for confronting what ML is poised to do. Which is why I was hoping for something a bit more general and constructive, but alas this isn't it.
https://wn.rudolfsteinerelib.org/RelArtic/BlackDavid/DB1981/...
Seems the author does technical due diligence at a VC firm now:
https://www.blackliszt.com/2021/10/what-is-technical-due-dil...
Well this skewers the worst tendencies of the HN comment section…
The digital revolution, which began some 30 years hence, has little to do with the more recent advancements in AI/ML. The reason digital technology feels so revolutionary is because it is - it's our first glimpse into a post-scarcity future. Distribution costs for a digital work to all of humanity is low, and approaches zero if we get rid of copyright.
Copyright is an antiquated system, suitable only for a pre-digital age. We need a better system. One that incentivizes the creation of works while also admitting that DRM to make bits uncopyable is like trying to make water not wet. If instead every copy that was distributed over the Internet, including via Bittorrent paid the author back, suddenly it's not a problem anymore. There are other problems/details to be worked out, but the first step is in admitting you have a problem.
Both yes and no, in a sense.
https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
We are doing this because our individual incentives are not aligned with those of the group. Moloch is the god of misaligned incentives, of coordination problems, of arms races and races to the bottom. It's a property of any system with selfish actors, which means any system that humans are part of, and probably any intelligent being that arose out of natural selection would have been hypothetically been part of.
From `Meditations on Moloch`:
> There’s a passage in the Principia Discordia where Malaclypse complains to the Goddess about the evils of human society. “Everyone is hurting each other, the planet is rampant with injustices, whole societies plunder groups of their own people, mothers imprison sons, children perish while brothers war.”
> The Goddess answers: “What is the matter with that, if it’s what you want to do?”
> Malaclypse: “But nobody wants it! Everybody hates it!”
> Goddess: “Oh. Well, then stop.”
Animus against that catalyst is so much hormonal groaning.
Human nature is constant.
