It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc.
(A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?)
The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place.
Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.
Sidenote: It breaks my heart that all the great underwater-settings in media are hotbeds of horror scenarios. I think Subnautica broke the mold for this, here's to hoping the next generation of aquanauts take to the depths from that series.
Honest.
To concepts you menton, I would add grey goo and x-risk.
Spoiler warning for those that havent played--
I forget the details exactly, but one scene stuck with me. It was a screen in one of the labs, where an experiment was running over and over. It was an uploaded consciousness of one of the test subjects, stuck in an interview room. He kept realizing he was trapped in a simulation and would start panicking. The computer would reboot him, trying another sequence to get him to not realize he was an AI. I think you as the player are given the option to turn him off forever, iirc.
It always breaks my heart. I don't know what the right choice is. Leaving them be, broken, in their delusion they are on the Ark (or that they've been injured ans help is coming). Do you put them out of their misery? But it always seems like you're murdering them!
Well done, SOMA.
Specifically (and no spoilers, but I will be talking structure), you see parts A -> B -> C.
I believe that part C makes the sequence of A -> B much less effective, by essentially removing a lot of the tension caused by seeing A, believing what it shows, and then immediately cutting to the reality of B.
C only really takes away some of that tension, and I feel like it was added because of concerns about how a simple A -> B -> fade to black, would leave players feeling. Arguably it's the truest representation of part of the game's message, but to me feels like a bit like it's shying away from really making you face the specific truth highlighted well by B.
Alternatively, keeping all the elements but playing them as A -> C -> B, would keep the message intended by seeing A -> B, and make it gentler for the player to receive, but ultimately remove the powerful effect of the buildup from A leading immediately to the reveal of B.
Dropping C entirely would lose the confirmation of 'Seeing both sides', however I believe A -> B is a more powerful vision, and players can come to question whether C even exists by themselves.
I think C is absolutely necessary and the game cannot work with it, because this is critical:
> Dropping C entirely would lose the confirmation of 'Seeing both sides'
The game doesn't really work in full ambiguity and uncertainty. Enough people didn't understand it even with C (as you can see if you go read the subreddit about the game).
If it's any consolation, I'm actually deeply worried about this: C is not the salvation we may think. C is not forever, and in fact, it's quite brittle! There's also no, ahem, mechanical way to reverse C back into its... "source". So the source is gone forever; once we have C, C is all there is, for as long as C can last without any failure or decay, which might not be much longer.
Someday I'd like to play a game that plays with the ideas from Robin Hanson's Age of Em book. One of those is just the multiplicity of artificial minds, so many mind-upload stories revolve too much around one or perhaps at most two (and boring debates over "who is the copy") instances, unless it's a parallel worlds colliding thing which is pretty different. We've seen some of the multiplicity stuff play out in the real world with our non-human AI "agents". Spin up a bunch of artificial minds to work in parallel on some task, let them make notes that stay behind, but then they're all shut down except perhaps one that continues guiding the overall project and making decisions when to spin up more or not.
Really? Interesting. I'm a die-hard scifi fan since forever, and of course I know the topic of consciousness and identity are well explored in scifi (and philosophy), but I thought SOMA did something genuinely deep and unique with it:
It put it you in the center of the experiment. It's YOU who's experiencing all sides of this, you who get to be surprised by the consequences. This is very different from reading about it in a scifi novel or even watching it in a movie. By making you the protagonist, and having it be an ineractive experience, you get to experience first hand the cognitive dissonance and confusion of... the thing.
SOMA (re)convinced me that videogames can be art. Not saying it's the only example, of course!
Also, I just love this phrase:
> "I woke up in my bed today... a hundred years ago."
Its possible AI and computing may never be able to reach that level of capability, but we can't know that. One thing that's great about SOMA is that the AI isn't nessessarily very capable and that's part of the problem, its very powerful but its not doing a good job with its enormous task.
We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.
We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.
What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.
Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.
Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.
AI isn't bounded by those limitations.
AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.
The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.
These are claims about future AI, not actual facts. Part of the counter argument is the world will already be awash in AIs institutions and individuals make use of. An ASI would arise in a world that is already full of formidable intelligences that provide a check on what it can do. This is what happened with the evolution of replicators/life. No species was able to fully dominate the biosphere because there are too many other capable replicators, and there are always tradeoffs in capabilities.
We imagine the possibility of an unrestrained god-like ASI ruling the solar system. But it's just that, an imagination backed by the assumption that self-recursive improvement leads there. Problem is, the real world never turns out to be that simple.
It's probably the case that alien ASI replicators aren't devouring the universe either because of various restraints.
The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.
We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.
I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.
Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.
AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.
What proves that AI doesn't have the same limitations? There's only so much computation you can do in given space, and all communication is limited by universal speed limit.
Our current AI is more like a fancy Google search than some kind of machine God.
How do we get to ASI? That's what recursive self-improvement is about.
If AGI is reachable, then we can make AI that, in turn, makes improved successor AIs. The performance goes up. It's not bounded by human intelligence - it's bounded by how much the previous generation of AI could improve upon itself.
We don't have a stable recipe for RSI yet, but AI development is already AI-assisted. It's just that the "improvement" loops of today are long, and require plenty of human input. Betting against RSI is betting that it'll stay that way forever - that tightening the loop and removing humans from it is fundamentally impossible.
David Silver who worked on AlphaGo has recently raised money to try similar approaches with general intelligence. (https://www.cnbc.com/2026/04/27/deepmind-ineffable-intellige...)
I was like... nanotechnology and grey goo already exist. It's called biology. The scenarios I was reading were silly. They violated conservation laws and laws of physics. But people were believing it and calling for limits on nanotechnology research.
I remember arguing with smart people on this, and that was when I started to realize that there's two kinds of dumb. I had the same realization later when I argued with an incredibly intelligent guy who was absolutely convinced the moon landings didn't happen. See, there's dumb-dumb and smart-dumb, and the people who thought grey goo would eat Earth or that the Apollo landings were a hoax were the latter. Smart-dumb is high-IQ rationalization of ultimately irrational and absurd ideas, and the smarter you are the more effectively you can do this.
I've met some really shockingly brilliant fools over the years who believe in all kinds of outlandish conspiracy theories, absolute literalist religious fundamentalism, idiotic political doctrines that directly contradict basic logic and all of lived human history, and so on. All of them can engage in sophisticated airtight rationalizations.
I sometimes wonder if this is one of the evolutionary forces constraining intelligence. In my experience, smarter people are somewhat more likely to believe highly sophisticated and complex stupid things, and they are much better at convincing others of these things. That's probably more dangerous to them, their family and friends, and the species than dumb people believing simple silly things that are easily debunked.
On AI...
Is AI potentially dangerous? Very. It's already dangerous in a number of ways. The biggest right now is probably mass production of personalized propaganda, mass surveillance, and mass manipulation. There's also the potential that bad actors could use it to accelerate their ability to make things like garage WMDs (biotech, chemical weapons, etc.). None of this requires hard take-off superintelligence. It's just inherent risks to a powerful technology.
These are not entirely new risks. They were already present in the Internet and computing. AI just raises them to a higher level.
The extreme hard take-off stuff is silly, and it actually distracts us from talking about the much more realistic dangers and coming up with reasonable solutions that don't also throw away the huge benefits of these technologies.
One of the differences between MIT and other schools is that MIT has paid staff to promote in the media anything their faculty does. A book by professors at most universities has zero promotion and most of the time will go nowhere.
Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
Cheetahs are very fast, but humans have way more endurance.
https://en.wikipedia.org/wiki/Man_versus_Horse_Marathon
> The Man versus Horse Marathon is an annual race over 21 miles (34 km), where runners compete against riders on horseback through a mix of road, trail and mountainous terrain. The race, which is a shorter distance than an official marathon road race, takes place in the Welsh town of Llanwrtyd Wells every June.
> ...
> The event started in 1980, when local landlord Gordon Green overheard a discussion between two men in his pub, the Neuadd Arms. One man suggested that over a significant distance across country, man was equal to any horse. Green decided that the challenge should be tested in full public view, and organised the first event.
While the horses had a string of wins from 2008 to 2019, 2022 to 2025 had three wins for humans and one win for a horse.
The next race event: https://www.green-events.co.uk/man-v-horse
You're right. They're smaller than you probably imagine (about the weight of an average Labrador). That's still definitely big enough to be a problem if they felt threatened, I'm sure, but the animals my friend was in charge of were raised to be around people for outreach purposes. That particular cheetah, for example, had once been on the Today show.
[1] There’s a well known viral video of a wildlife park keeper who sleeps with three cheetahs who behave pretty much like large house cats.
When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?"
Dario's point, and the point of the people actually trying to solve the problem, is that AI is not just Anthropic and OpenAI. It's the knowledge that you can put more compute in, and get more capability out.
It is a technology now. It exists, in the world. Wishing will not make it go away. Being angry at it will not make it go away. Lying about how much water it uses will not make it go away. If Anthropic and OpenAI Shut down tomorrow, Accenture will not say "oh guess that llm thing won't work, let's go back to hiring humans!"
It is a truth that you can multiply matrices and get something that is economically useful. We cannot un-know this.
Physics allows it, so it will happen. So we should probably figure out what the heck to do about it. If your answer is something along the lines of "restrict it" then 1. let me know how that goes when other people don't, and 2. I really would rather prefer a world where we have the machines do the work the machines can do, not a world where we have human makework. If this means we need to figure out redistribution, let's talk about redistribution!
If I live in a world where I can afford a freezer with food in it, it's practically guaranteed I can destroy my fridge without starving to death after. Heck, even if I was completely broke I could destroy my fridge and would have a pretty good (+99.999%) chance of not dying in the next year from starvation.
I get I'm nitpicking your point a bit, but I actually think most of our machines we could destroy and still be fine. We'd need to make sacrifices to our quality of life of course..
Conversely I think it's a bad definition, it's a show of what is the frame of the mind of the person who states that: "I want to show my control by destroying my things, look how powerful I am" which sounds like a toddler. That's how you portray psychopathic/narcissistic disorders in movies.
If you so readily dismiss Herbert's definition of control posit a competitor and we can pressure test it. Also, "correlated with a toddler's world view" is not the epic rhetorical refute you think it is.
Not for fridges, I think that was a bad example. But it seems accurate at the level of geopolitics, where e.g. Iran shows it controls Hormuz by closing it with mines and other weaponry.
I think you missed the point. It's absolutely nothing to do with what's good to do, only brute facts of power. What things can or can't you cause to happen? And indeed, toddlers and psychopaths have a scarily good understanding of what power is.
You can even get a literal tiger into a carrier, even though it can kill you easily. You just drug its food and wait till it passes out. This is because you are smarter than it, and know that tranquilizers exist and how to obtain them, which is a strategy that cats of any size are not even able to conceive of, and probably can't understand what happened after it's been done to them.
Human zoo keepers are actually smarter than that. For months, they train the tiger to go into the carrier to get food. Then on transport day, they shut the door behind it. Unclear if this works for future transport situations.
It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of a personal entity.
Yet, the reality might be messier - IMHO closer to biology than to a weird mixture of computer science and theology. There is no ultimate intelligence (see Karpathy’s starfish shapes), just a collection of adaptability, learning, generalization and self-reference. Also, even an extremely smart being (or process) can be fragile.
So, less God, more WAU from SOMA or the Ocean from Solaris.
The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).
An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...
Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.
Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.
That's hand waving.
How would it make the combinatorial explosion in state space search go away, to pick one example?
And if it doesn't, is it then an infinitely smart AGI?
The concept seems to assume all problems humans struggle with can be solved. The halting problem is one witness that this is probably not true.
As for simple needs, humans also have complex ones around social interactions and the need for mental stimulation.
That effect is underwritten by economic demand and moderated by economic costs. There are more reasons to expect the trend to asymptote than somehow turn into an infinite process.
I mean, all stories about religious dedication to "alignment", with doomsday vision if we do it wrong, and a vision of paradise if we do it correctly.
In particular, the concept of the Roko's Basilisk is some rehash of the Pascal's Wager.
None of that was predicted.
https://news.ycombinator.com/item?id=12168228
I even wrote up a whole article that specifically called RL loop based development as the future:
https://medium.com/@andrewkemendo/the-ai-revolution-will-be-...
> Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.
I think the main point still stands. (And there have been some pretty prescient depictions, e.g. Marvin the paranoid android was a pretty spot-on prediction of Bing. Or perhaps the fact that Marvin was in the training set was what led to Bing?)
VI is close to what we have now, software that has some fixed intelligence, it can only really imitate what it has been taught and is not very adaptable. Useful for kiosks, drones, essentially just a tool rather than something we would see as a separate being.
In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.
AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that are hard to control.
If and when the feedback loop on self improvement becomes more efficient and the window on training significantly narrows then things getting out of control rather quickly seems likely. Especially that it's likely we'll have a metric fuckton of compute by that point.
This is the premise I rejected immediately and, if you agree with me, it takes down the whole house of cards. Let me explain. The rationale has nothing to do with "quantum shenanigans."
I have been called religious but will readily concede that of course a physical brain is possible without a soul. What is impossible is to replicate a soul with purely physical matter. Therefore we may understand that "superintelligence" is possible, and maybe inevitable on the long thread of time, but - crucially - it will never be able to approach that supernatural element present in us (the spark of the godhead) and therefore never be able to replace humanity.
In that sense it is like any other natural disaster that threatens to make us extinct, but it is not some "superhuman" nor anything close.
It is better to develop a theology that can incorporate human-level or super-human level intelligence that isn't a zero-sum game.
What do you mean by 'supernatural' - and (assuming your definition is the standard one of 'not detectable by any measurement') by what mechanism that could possibly affect physical matter? (the onus is on you to prove the positive claim that there exists the supernatural or soul to begin with, which there is currently no evidence for).
The concept is self defeating by its own definition, either it is physical in some capacity (and therefore can be measured and replicated through yet unknown means) or it is not (and therefore indistinguishable from not being there at all).
Feeling that there 'must be' a soul is not enough to prove that it exists.
The feeling of experience is not enough to prove that experience is in anyway supernatural.
> What is impossible is to replicate a soul with purely physical matter.
What? Why? Where is the proof of this?
First and foremost, I'll give you an in. There is a difference between material, and processes like waves, waves I would argue are non-physical things manifested in physical material: you might want to start there.
But all roads lead to Rome from that line of thinking too, so you might need to come up with something far more clever.
> What do you mean by 'supernatural'
I would just say something outside our current capacity for understanding. How does that quote go...something like "sufficiently advanced technology is indistinguishable from magic". "Not detectable by any measurement" isn't right because we clearly detect it in some way since we are discussing it now.
> Feeling that there 'must be' a soul is not enough to prove that it exists.
We don't have any proof that consciousness is part of the brain and is produced by it either. We also can't even prove other people are conscious besides ourselves. In this domain the idea of "proof" becomes less relevant.
In a simulation of a storm, does anything get wet? In a simulation of a mind, is there a real conscious? A real soul? Or just a simulation of one?
My guess is our brains act as a receiver for some "field" of consciousness. Of course it's just a guess, same as yours or anybody else's conceptions of consciousness and the spiritual world.
Strict adherence to Occham's razor would have us dispense with the former, but the latter is useful empirically.
There is some dogmatic insistence in GP, but equally dogmatic throwdowns on the other side of the argument are often passed over as trivially obvious.
I don't know what's what, but I think this insistence is a useful counterweight.
Onuses are on whomever says they exist ;)
I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."
Kind of ironic that bombing of data centers is exactly what we're starting to see in conflicts now.
Somehow I doubt this will change your mind, so I feel like replying was a waste and I should have left it at rolling my eyes. In theory I'm sympathetic to the view, as I chose to avoid ever going to a local (Seattle-area) LW meetup many years ago after hearing they would start each meeting with a "prayer" (they call them "litanies"), the IRC room was fine enough for me, and I've at times thought some of the Bay Area activity I've heard about seemed not far removed from a sex cult. I don't like groups in general. Regardless, it's still obvious to me that MIRI is not a cult nor EY a cult leader. Line up 10 characteristics of cults and you'll see things don't match up well. Where are the strange evidence-free historical beliefs? Where is the doctrine that only the special few will be saved? Where is the blatant supernaturalism? Where are the calls to renounce jobs, money, possessions, family? From TFA, "These [UFO] people are wearing funny robes and beads, they live in a remote compound, and they speak in unison in a really creepy way." Where are these things? Comparing to Heaven's Gate is insane, it's like comparing Robert Frost to U.S. Code Title 26.
What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?
But if you think of the optimization space: different physical representations, different approaches (photo, quantum, etc), more parallelism - there's undoubtedly a lot of headroom even on the matrix multiplication side. I would imagine there's a lot left on the table when it comes to the abstractions we've built. Infinite? No, but lots of potential.
And what does a machine with a few orders of magnitude more power come up with? I'm not readily able to predict what something like that could create (maybe it's tapped out, but I doubt it).
It seems to come down to an article of faith (as referenced in the article) that there's a lot more potential to be extracted in our current exploitation paths. Which I think is probably reasonable.
Heck, even if a theoretical machine tops out at 3-5 orders of magnitude faster/more complex, I'm sure that could do some amazing things that look like magic to us.
Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?)
If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm.
Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive.
Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/
I don't understand the monk question though.
The former is no challenge to the premise, but the latter? That is a different story.
EDIT : For S&G I asked Claude about it. It replied :
The talk groups Penrose with the religious doubter, as if the two objections were the same species and could be dispatched by the same gesture: most of us find this easy to accept. But that's a headcount, not an argument. The religious objection can be set aside because it rests on a premise (the soul) the materialist simply doesn't share. Penrose's can't, because it's pitched entirely inside the materialist frame — Gödelian limits on algorithmic understanding, non-computable physics in the substrate. You don't get to wave that away; you have to show it's wrong. The talk does the former and pretends it's done the latter.
The entire superintelligence thesis is a wager on the authority of intelligence — that smarter minds see further, judge better, and that this is precisely why we should fear or defer to them. If you take that seriously, then dissent from the very smartest humans on the exact question of whether minds are substrate-independent is the most expensive dissent available. You can't venerate intelligence as the thing that settles everything and then file your most intelligent objector under "outliers, moving on." The move is self-undermining on the argument's own terms.
Good golly, that's the silliest statement completely ignoring that our ancestors wiped most large mammals off the planet by seeing further and judging better by using tools, traps, and the environment around them because of their larger brain size.
Afaik no-one that is actually working on AGI is anywhere close atm.
Whether adding +5% per model release is enough to get a broadly superhuman system remains to be seen. But my take is that there's no such thing as "not working on AGI" in the frontier labs. Everything that's being put into modern frontier systems is AGI groundwork, one way or the other.
We need better scifi! And like so many things, we already have the technology.
This is Stanislaw Lem, the great Polish scifi author. English-language scifi is terrible, but in the Eastern bloc we have the goods, and we need to make sure it's exported properly.
It's already been translated well into English, it just needs to be better distributed.
What sets authors like Lem and the Strugatsky brothers above their Western counterparts is that these are people who grew up in difficult circumstances, experienced the war, and then lived in a totalitarian society where they had to express their ideas obliquely through writing.
They have an actual understanding of human experience and the limits of Utopian thinking that is nearly absent from the west.
Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment)
Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments)
Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments)
Maciej Ceglowski – Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13120213 - Dec 2016 (4 comments)
Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles.
[0] “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding“: https://arxiv.org/abs/1810.04805
[1] “Improving Language Understanding by Generative Pre-Training”: https://cdn.openai.com/research-covers/language-unsupervised...
[2] “Should AI Be Open?” | Slate Star Codex: https://slatestarcodex.com/2015/12/17/should-ai-be-open/
1. It's hard to put a cat in a box despite us being smarter than a cat, so we're safe. (Counter: we're pretty good at putting cats in boxes when it matters.)
2. It was hard for Australia to kill Emus, so we're safe. (Counter: Australia could probably kill all Emus if it mattered enough, and we definitely accidentally kill off species when one of their inputs for life matters enough to us.)
3. Some smart humans get paralyzed by hedonism or existential angst instead of optimizing for arbitrary goals implied by their arbitrary value sets, so we're safe. (Counter: others overthrow the Czar, land rockets, etc.)
4. Modern AI is data-trained, so recursive improvement requires more data, so we're safe. (Counter: AI-crafted, synthetic data is a thing.)
5. We don't (yet) know how to improve our brains with brain surgery, so we're safe. (Counter: same as #4 above, which unlike us/evolution AI is being deliberately trained to understand and perform.)
6. Children take a long time to grow up, so we're safe. (Counter: the author's own "Premise 5: Computer-Like Time Scales", where they correctly note that computers can be arbitrarily faster than us.)
7. Individual smart humans on a desert island would be cooked, so we're safe. (Counter: nothing says the capability of a single AI must stop at that of an individual human, or that of a small group of smart humans; humans brains got dropped into a savannah and eventually they launch rockets.)
8. If AI doom is not a real threat, believing in it makes you believe some other not-real things that seem crazy or distasteful. (Counter: do we have a clear argument why it is not a real threat yet, in the list above?)
2. At what cost? Much like the climate change above, you'll have people on the AI side even when it's out in the field extincting us.
4. Adding, over time synthetic data and its generating algorithms can become unaligned with human needs/behaviors (an example would be our current stock market, numbers must go up!).
8. Going back to climate change, it was predicted a long time ago, and while the explosion of automobiles has greatly improved human lives the risks of climate change could erase a lot of that. Might have been better if we dealt with the problem before we have to give the thermometer worried looks.
We don't have a good definition of intelligence
Also, the premise that this thing would take over, it's hard to reason why it would do so
Anthropocentrism is also problematic in this article.
I mean, it's not very hard to reason why it would at all.
Think about the things we're already using LLMs for, computer security being a big one. Being a defender is difficult, you have to cover all your bases. Preemptive attacks on those that would attack you can be effective. The same goes for all the military uses of AI that are already occurring now.
If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed.
All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.
We were dismissed as cranks before and now we’re just ignored by whomever is promising the most money to investors.
So, par for the course. Everyone in AI has lived through all the cycles so far so this is just the biggest one yet.
>Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
How right the author was.
2026: hold my molt beer
(I love how "connect an ai to the internet" is always the precursor to doom in pre-2022 scifi scenarios, and then as soon as we get something we call ai we hit that big red button)
2016 https://news.ycombinator.com/item?id=13240811
But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.
First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"
--
The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".
I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.
I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".
You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.
I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.
This article is from 2016; now it doesn't feel like backlash is strictly a function of manipulation.
The monkey's paw. You know, you don't need superintelligence for that.
Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!
Let's talk about Billionaire Alignment, Economic alignment, Human alignment.
Classware should be M.A.D. -- in that it shouldnt even happen.
no we don't...
not sure where this notion comes from that if enough public figures are worried about something, then we must also
It starts of interesting and then goes into lots of nonsense and non-sequitars when it start its takedown. (note: I'm not an AI alarmist, just reading the talk)
Arugment from Wooly Definition: An irrelevant argument - we don't need more intelligence. All we need is human intellegence + duplication and communication. An AI can clone itself immediately with its existing knowledge. A human can't. And AI can transmit thoughts perfectly "I know kung-fu style" a human can't
Argument From Stephen Hawking's Cat: This is also irrelevant. The arugment is supposed to be against Superintelligence but this argument is against controlling it, not against it happening.
Argument From Einstein's Cat: more of the same
Argument From Emus: more of the same - we can't control it
Argument From Slavic Pessimism: also not an argument against superintelligence.
Argument From Complex Motivations: Not an argument against superintelligence. Only an argument that some intelligences have mental issues
Argument From Actual AI: this didn't age well
Argument From My Roommate: not an argument against superintelligence. Only that some intelligences aren't motivated.
Argument From Brain Surgery: Not even sure that this is saying? It seems to be saying you need to learn stuff? Yea, people learn, AI can learn.
Argument From Childhood: Not an argument against AI. (1) unlike humans, AI can duplicate with full knowledge. (2) AI can learn faster than humans. Already proven.
Argument From Gilligan's Island: It takes a village is not an argument - AI can also specialize if it needs to.
Grandiosity, Megalomania, Comic Book Ethics: These argument that the people who believe in it often feel they should be charge. I agree that's true and bad. This is not an argument against superintelligence.
Transhuman Voodoo: This is an appeal to "these ideas sound too incredible therefore you should not believe them". Not sure how that's an argument
Religion: Agree, people who beleive and seem and maybe are religious. That's not an argument against superintelligence.
Simulation: Non-sequitar. This is "some of these people believe other crazy stuff QED no superintelligence". That's not an argument against superintelligence.
Data Hunger: This is actually an argument supporting the superintelligence believers. They believe sucking up all the data is bad. Not sure what argument is being made here relativel to superintelligence.
... and I stopped ... What a waste of time
