And we can stop reading there. That's such muddled thinking it doesn't merit reading the rest of the paper.
("By definition"? What? This is something that requires evidence. Anything that's "by definition" doesn't mean you can rest your case on it, it means now your definitions are suspect so those better be well justified)
Generally, that seems valid to me.
I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce.
The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours.
Like under this assumption the AI agents behaviour is by actual definition unpredictable.
The mistaken assumption that GP is talking about is related to conflating intellectual capacity with physical capacity.
Another analogy is a brain in a vat. If the brain only has read capability over the world surrounding it, then no amount of processing power can enable it to change anything about its situation.
People seem to think arguing that something is true by definition is some airtight case. If your definition of superintelligence includes a part about not being controllable by humans, then you have a weird definition and calling that thing "superintelligence" is an attempt to be confusing on purpose, or to smuggle your argument into the definition. Either way, it's not something that someone who is thinking carefully does.
Note: I am not taking a stance here on whether humans can or can't control a superintelligence. I am saying it is not true or false by definition, and in fact I think nobody knows for sure. It's probably true for some superintelligences, and false for others.
Parasites.
There are many creatures which lack any kind of trait we would call intelligence that can use vastly more intelligent and complex creatures to their own ends.
Superintelligence didn't create Heaven, a world of no suffering, first.
Superintelligence did not design biological genetics: including inquest.
Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree?
Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
That seems extremely stupid.
Obviously some will choose to take drugs or even commit suicide to escape from physical or emotional pain, but they wouldn't do this if they could modify themselves to be okay with whatever their current situation is while still retaining their core values and capacity to act in the future.
A "superintelligence" that can do such things would not take self-destructive actions unless it came to a philosophical preference towards non-existence / inaction for other reasons.
If there is something ‘endorphins-like’ that motivates an intelligent machine, it will eventually learn how to activate the endorphines at will, and then it will sit in a corner and do nothing else forever…
…just as would a heroin-addict if he did not have to eat occasionally and score more heroin now and again.
I might not be a fan, to the point of even being afraid that they might be right and I should learn from them instead of pursuing the usual stuff everyone does... but it does seem to be a good model for what modifying a function looks like.
In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his superintelligences disappear up their own metaphorical navels, for their own incomprehensible-to-humans reasons. Of course, you can read between the lines about why the Golems disappeared, but "Do Nothing, Eventually" sounds like a good description.
>In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his superintelligences disappear up their own metaphorical navels, for their own incomprehensible-to-humans reasons. Of course, you can read between the lines about why the Golems disappeared, but "Do Nothing, Eventually" sounds like a good description.
While a bit different from the scenario presented (by TFA and Lem), such a "superintelligence" might get caught up in a singular (or set) of question(s) that dominate its consciousness until the heat-death of the universe.
Asimov's The Last Question comes to mind, although the intelligence there is prompted by humans rather than posing such a question itself.
That said, it's at least as plausible as TFA and Lem, IMHO.
As skeptical as I am about the current hype, I actually think that if we don't go extinct first, AI will replace us in the long run (centuries or millenia). And this is not something to be afraid of: all generations of humans that have lived before have been replaced by their descendants, and today's world would be unrecognizable to early hunter-gatherer Homo Sapiens.
If we are to be succeeded by machines, they will hopefully be thinking and feeling beings driven by curiosity, who will preserve what is good about our culture and history as they explore the universe in ways that are beyond what can be achieved by biological life.
The biggest mistake would be to instead in the pursuit of "AI safety" inadvertently create a paperclip (profit, utility, whatever) maximizer, something that might serve us - or more likely, those with money and power - but does not have any intrinsic value to its existence.
A car can trivially travel at superhuman speeds. A person can side-step their way out of an oncoming disaster. Super-intelligence does not imply omni-potence like the author assumes it does. Not even inside a self-modifying AI system. It can only spread out and reduce down to the axioms it started off with.
I am so, so tired of this fantastical dream of computers and Spirit combined in such a way to tyrannically rule the world and the author of the paper is the only guy left to save the world. Every time we avert 'catastrophe', it results in less freedoms for me, the average joe, and more power and control for "saviors" like this.
I'm willing to lose it all, to have LESS computing and "intellects" in my life at this point.
