undefined | Better HN

0 pointsfasterik3y ago0 comments

There are plenty of scenarios we should be thinking about. I'm mainly concerned about the logical leaps and scant evidence used to go from "scenario X is possible" to "it's more likely than not that scenario X will happen, therefore we should ban all research".

>Then our fate is out of our control, which might as well be death.

I disagree with this part. Dogs aren't in control of their fate, but a lot of them seem to live pretty good lives. I think the ethically important thing isn't being the dominant species, it's getting what you need in order to live a meaningful life. Humans need food and shelter, social connection, creative projects, entertainment, art, etc. These things aren't inherently threatened by AGI.

0 comments

2 comments · 1 top-level

NumberWangMan3y ago· 1 in thread

People keep dogs around because we have evolutionary instincts that find them cute, and we also have an evolutionary need for companionship, neither of which really applies to an AI. We also need biological food, which means we keep plants and animals around in general. And still, there have been plenty of cases where we wanted something, like oil, and we got it by killing all the whales. We wanted land for farming, and destroyed many, many entire species by cutting down forests. We keep cattle around just to eat them, and once we get artificial meat right, probably the number of cattle is going to drop drastically.

This is humans doing this, and we're not even that smart, or that relentless. We get tired, we get "good enough" and stop, but we don't know how to build an AI that just tries to do the simple thing and stops. I mean, there's even the known problem that an AI who only tries for good enough will itself likely end up designing an AI that tries really hard. Look at us--it's what we're doing!

Can you imagine humanity, but 100% focused on maximizing profit, with everyone working around the clock to produce, produce, produce? Can you imagine trying to write laws to cover every possible externality, every possible way that people might pollute the environment, consume too much natural resources, or screw up the things that capitalism doesn't cover?

The fear with AGI is that the lessons we learn trying to align lesser AIs (which we are not even close to being able to do) will not generalize to higher levels of intelligence, which are coming very quickly. Alignment is a very hard problem, like putting a lander on the moon, and right now we're at the stage where the people working on these AIs are trying to launch rockets and figure out the equations for trajectory while it's on its way. "We'll try to have the first AIs we build align the later ones" is an actual thing that people think is a legitimate strategy. And I agree with Eliezer Yudkowsky -- it would be fine if each mis-aligned superintelligent AI we build would only kill a couple million people like a nuclear bomb, instead of being deadly.

Even folks like Paul Christiano who are more optimistic, but take alignment seriously, are not very optimistic: https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-ou...

I'm gonna be honest -- I tried learning more about what actual people who work in AI alignment think about this issue, hoping that I could find something to reassure myself, but I can't find anything that makes me think we have much of a chance. There's nothing special about humanity -- it's entirely possible that most of our paths into the future involve extinction. Maybe all paths do, eventually, but on the hope that there's some sort of future that isn't a paperclip universe, I'm pushing really hard to make sure everyone understands what we're in for. I may be wrong -- I really hope I am. If not, I'm sorry.

fasterikOP3y ago

>Can you imagine humanity, but 100% focused on maximizing profit, with everyone working around the clock to produce, produce, produce? Can you imagine trying to write laws to cover every possible externality, every possible way that people might pollute the environment, consume too much natural resources, or screw up the things that capitalism doesn't cover?

Sure, I can imagine a lot of things. I can also imagine a version of humanity that is a thousand times more wise and ethical than we are. Being able to imagine something doesn't mean that it is likely. I'm not convinced that the worst case scenarios significantly outweigh the best case scenarios and the benign scenarios in the probability distribution, because I don't think we have enough information to assign accurate probabilities to vaguely defined future events.