undefined | Better HN

0 pointstwo_in_one3y ago0 comments

It's not a mass destruction weapon. But that's not the point. You have to have good guy here. No other options.

0 comments

4 comments · 2 top-level

TeMPOraL3y ago· 2 in thread

The entire thing with worrying about GAI starts with observation that it is a WMD, so powerful we can barely conceptualize it.

A smart enough AGI, unless it's perfectly aligned, will end humanity (or worse - there are worse things than death), most likely by accident or just plain not caring. But it doesn't stop there. If that AI is self-improving, it could easily turn into a threat for the entire galaxy, or even the universe, unless it meets a stronger and better aligned (to its creators) alien GAI...

That's the alignment 101. But I worry people don't talk about alignment 102: a perfectly aligned superhuman GAI will not destroy us, but its very existence will turn us into NPCs in the story of the future of the universe.

malaya_zemlya3y ago

The fact that we still exist speaks strongly against an AGI destroying the entire galaxy. Fermi Paradox, otoh, suggest that planetwide destruction is not out of the question.

digging3y ago

Unless we're earlier than we think. Someone has to be first.

digging3y ago

As in all systems, it is a Bad Idea to begin a process that has extremely negative possible outcomes and hope that someone will do the right thing to prevent those outcomes.

j / k navigate · click thread line to collapse