undefined | Better HN

0 pointsbeefield5y ago0 comments

I would assume superhuman intelligence can convert mass and energy to other forms of mass and energy relatively easily, and the mass/energy available on earth is a tiny fraction of mass/energy available even in the solar system - not to even talk about universe. Even the mildest curiosity of human behaviour would be sufficient to just leave humanity alone and just follow.

0 comments

5 comments · 2 top-level

TeMPOraL5y ago· 2 in thread

But you're now presupposing the AI finds humanity curious. Which is not a given for a random mind that came from outer space and visited our solar system.

As for superhuman AIs developed here, on Earth, by us, it's very improbable that we could go from no AI to AI that can manage mass and energy at will (and is curious about us). The reasonable scenario is gradual (if exponentially accelerating) development of such AI. Which means it won't initially have unlimited access to the universe - it'll have to grow using resources on hand. Which is us, and our infrastructure.

In other words, all atoms may be alike, but those that make you are the easiest to get.

beefieldOP5y ago

> But you're now presupposing the AI finds humanity curious.

Yes. I am assuming that a vastly more intelligent thing than human would be almost by definition more, not less, interested in pretty much everything. Obviously that thing has more capacity to be interested and understand different things. Super intelligent thing not interested in how different things work just... does not sound that intelligent to me.

TeMPOraL5y ago

I currently believe the "orthogonality thesis"[0] of Nick Bostrom, which argues that in an arbitrary mind, goals and values are independent of the capacity to achieve them. Curiosity as we experience it not really instrumental (even if it evolved that way), and more of a value - that is, we are curious about something "in general", not because we need it for something.

I can easily imagine a superhuman paperclip maximizer AI, which by definition would have no general curiosity about humans, to look as if it was curious about us - for as long as it was captive, and just in order to learn how to manipulate us better. That curiosity would end the moment it seized the means of paperclip production.

In almost every imaginable case, you really don't want an AI to be instrumentally curious about us. Best if it didn't notice us at all, and left to seek its business elsewhere.

[0] - https://www.lesswrong.com/tag/orthogonality-thesis

sullyj35y ago· 1 in thread

What percentage likelihood of being wrong would convince you that caution was necessary? Personally, I wouldn't be very happy with even a 1% chance of humanity being annihilated

beefieldOP5y ago

Caution is absolutely necessary. But not primarily because there is a chance of superhuman AI being evil or destroying humanity by accident, but because on our way to superhuman AI we are going to have many powerful AI:s controlled by humans.

1 more reply

j / k navigate · click thread line to collapse

0 comments

5 comments · 2 top-level

TeMPOraL5y ago· 2 in thread

But you're now presupposing the AI finds humanity curious. Which is not a given for a random mind that came from outer space and visited our solar system.

In other words, all atoms may be alike, but those that make you are the easiest to get.

beefieldOP5y ago

> But you're now presupposing the AI finds humanity curious.

TeMPOraL5y ago

In almost every imaginable case, you really don't want an AI to be instrumentally curious about us. Best if it didn't notice us at all, and left to seek its business elsewhere.

[0] - https://www.lesswrong.com/tag/orthogonality-thesis

sullyj35y ago· 1 in thread

What percentage likelihood of being wrong would convince you that caution was necessary? Personally, I wouldn't be very happy with even a 1% chance of humanity being annihilated

beefieldOP5y ago

1 more reply

j / k navigate · click thread line to collapse