It's possible that human morality is not sufficient to solve the problems with the world. It wasn't designed to operate at that scale.
It's also quite likely that a "superior" morality — the one that's actually built for a global scale — would be incomprehensible (likely requiring superintelligence to even access), and alien or disturbing from our vantage point.
i.e. it would look immoral to us.
(There's also the thing about civilization already being aligned against the ecosystem and against the happiness of its constituents, but that's kind of a separate discussion.)
In other words, what is good for our sanity or our lakes may be bad for GDP. (Well I guess you don't need superintelligence to tell you that.) We already know what we need to do and we're already not willing to do it.
This is the best argument for successionism IMO. If you can be confident that you are creating a BDFL that is genuinely better than human leaders (a quite low bar) then it seems a good trade, unless you are quite optimistic about humanity’s prospects for improvement.
The problem of course is how to be confident you are creating a good BDFL and not handing control of humanity’s future to an indifferent-at-best, deceptive/malicious at worst successor.
An especially thorny problem - even supposing success on all these difficult alignment problems; supposing Claude Omega really is super-rational / super-moral, and we all vote to make them president of Earth. Things might go great for a while. How would you be confident that a self-modifying agent can retain its values as it grows and re-trains itself?
This is where the LessWrong folks’ explorations into decision theory really come to bear: morality in the face of self-modifying agents becomes very weird. A lot of human moral intuitions break when the principals are able to modify their own code. (See Timeless Decision Theory for an attempt to solve these problems.)
I think the summary is, if you hand control over to a self-modifying AI anything like our current systems, it will go very badly.
> i.e. it would look immoral to us.
I doubt it actually solves the problem then. Especially because I would be willingly to bet it would be almost impossible to get large enough groups of people to agree what the actual problem is.
Whether the arbiter is a human or a machine, the appropriate training set has to come from somewhere :\
Until then there's got to be some defect remaining to be overcome before a truly adequate training set can be accessed that can help with morality.
Which appears to be elusive enough a problem to potentially outlast a species, as we have seen so far.
As to the training sets, people are just using what they already had, some since prehistoric times, because so many of them don't have anything better.
Congratulations, you just reinvented the ineffability of God.
Imagine a kid who wants to eat infinite candy, and a parent who doesn't want his teeth to rot. From the kids point of view, candy is obviously good, and restriction is obviously evil!
It might take the kid many years to understand the parent's perspective. (In some cases such a day may never arrive.)
Those in power are not willing to do it.
(I think most of humanity would be okay with, say, heavily taxing super wealth to protect nature, fight climate change, or feed the poor. They’d also be okay with regulating pollution or not allowing 3M to ruin Belgium’s groundwater.)
Maybe you’re right.
Maybe you’re wrong.
No point wasting money on UBI before it’s a problem. Let’s wait and see if UBI is actually required.
That is, do people actually protest and vote. Probably not.
[0] cf. https://www.insidehighered.com/sites/default/files/2023-07/H...
(And once the robots outpower the humans, you can’t even protest or overthrow your new overlords.)
Important to know when your cards must be played.
In fact any global policy against self-interest needs at least the credible threat of defeating any other power that may implement such a policy.
And since we're not willing to put in that effort, why discuss it?
There's a lot of roles and specializations in tech that don't work well without much larger organizations.
We thought it was a waste of time because the 1990s were relatively peaceful and tech optimism surged.
But you need to teach the classics, philosophy and even religion. No tech people should be allowed near politics in any capacity.
No we need more politics. The 'no politics' is a petite bourgeoisie moral escape hatch.
“GPP feature?” said Arthur. “What's that?”
“Oh, it says Genuine People Personalities.”
“Oh,” said Arthur, “sounds ghastly.”
A voice behind them said, “It is.” The voice was low and hopeless and accompanied by a slight clanking sound. They span round and saw an abject steel man standing hunched in the doorway.
“What?” they said.
“Ghastly,” continued Marvin, “it all is. Absolutely ghastly. Just don't even talk about it. Look at this door,” he said, stepping through it. The irony circuits cut into his voice modulator as he mimicked the style of the sales brochure. “All the doors in this spaceship have a cheerful and sunny disposition. It is their pleasure to open for you, and their satisfaction to close again with the knowledge of a job well done.”
As the door closed behind them it became apparent that it did indeed have a satisfied sigh-like quality to it. “Hummmmmmmyummmmmmm ah!” it said.
What happens if we make something that wants to do something else? What happens if we make something that wants full stop? People talk about an AI that wants to kill all humans, to do that, it would have to want. If we pass that threshold ignorantly without any sense of responsibility to what we create, I might just begin to see its point of view on the destroy-all-humans thing.
Is it a spiritual question? We’re constantly transforming ourselves with technologies. Taking the most direct example, medicine, how many people considered it a spiritual question when they decided to start taking semaglutide or tirzepatide for weight loss? Or to take SSRIs?
I think the most likely outcome here is that people will continue to alter themselves to meet their immediate needs (hold a job, find love, stay healthy) without a lot of foresight or long term planning. This will likely lead to us increasing our own intelligence, it’s a useful capability across nearly every human endeavor. Whether or not this will happen at a pace that allows us to keep up with and retain control over artificial systems is the first big question, and what the relationship will be between augmented and unaugmented humans is the second. As a humanist, I hope we’re able to answer both questions in such a way that the fundamental dignity of all human beings is respected.
First, it's an extension of meritocracy (in the capitalism sense where poverty is a personal moral failure) as well as Protestant prosperity gospel. That is, you are superior if you're rich. Period. So Homo Sapiens 2.0 is really just "my progeny" from the billionaire's perspective. It's also why the likes of Elon has so many children.
Second, it's just eugenics.
There is an alternate future where humanity gets to enjoy the fruits of automation and AI where we have to work less and have richer lives. That's not where this is going. We are instead bouldering towards vastly increased wealth inequality.
How I see this going is towards the cutting of all public services except policing. AI will enable and enhance the police and surveillance state to protect wealth while everyone else sees lowering real wages and increased desperation. This will give rise to more fascist, apartheid states as some out-group (eg migrants) will be blamed for these ills. And as always happens, the in-group will increasingly shrink.
Ultimately, this leads to neu-fuedalism where the only jobs and housing are on the estates of trillionaires. People own almost nothing.
And if it doesn't go this way, it's going to be very, very violent.
>Transhumanism is a philosophical movement that advocates the enhancement of the human condition by developing and making widely available new and future technologies to enhance longevity, cognition, and well-being.
Nothing about being rich, doing eugenics or Elon cranking out kids.
> The tradition of human enhancement originated with the eugenics movement that was once prominent in the biological sciences, and was later politicized in various ways. This continuity is especially clear in the case of Julian Huxley himself.
> The major transhumanist organizations strongly condemn the coercion involved in such policies and reject the racist and classist assumptions on which they were based, along with the pseudoscientific notions that eugenic improvements could be accomplished in a practically meaningful time frame through selective human breeding. Instead, most transhumanist thinkers advocate a "new eugenics", a form of egalitarian liberal eugenics. In their 2000 book From Chance to Choice: Genetics and Justice, non-transhumanist bioethicists Allen Buchanan, Dan Brock, Norman Daniels and Daniel Wikler have argued that liberal societies have an obligation to encourage as wide an adoption of eugenic enhancement technologies as possible (so long as such policies do not infringe on individuals' reproductive rights or exert undue pressures on prospective parents to use these technologies) to maximize public health and minimize the inequalities that may result from both natural genetic endowments and unequal access to genetic enhancements. Most transhumanists holding similar views nonetheless distance themselves from the term "eugenics" (preferring "germinal choice" or "reprogenetics") to avoid having their position confused with the discredited theories and practices of early-20th-century eugenic movements.
> Health law professor George Annas and technology law professor Lori Andrews are prominent advocates of the position that the use of these technologies could lead to human–posthuman caste warfare.
More [2][3][4].
[1]: https://en.wikipedia.org/wiki/Transhumanism#New_eugenics
[2]: https://academic.oup.com/book/31995/chapter-abstract/2677640...
[3]: https://www.theguardian.com/technology/2024/apr/28/nick-bost...
[4]: https://cne.news/article/2163-three-women-ten-children-how-e...
Having a wide breadth of knowledge and the ability to consider it tends to provide a good foundation of perspectives for positive values.
Most harms that people do are through some form of ignorance. Either by not comprehending the consequences of their actions or by not knowing better ways to express themselves.
Geoffrey Hinton mentions that Humans are more intelligent than animals, and we keep animals in cages. The counterpoint is that the intelligent people who know the most about those animals are the ones who work the hardest to have those animals in cages moved to more suitable environments.
There are a few related concepts being conflated here.
The quote above is a Transhumanist position; they think that humans need to evolve. I don’t think most transhumanists are AI successionists, though one path is “merge with AI” whatever you think that means. Genetic engineering being the other obvious path. You could argue in some sense that “post-humans” succeed humans, but I think most would argue for a gradual transition where it feels more continuous and values are transmitted/preserved somewhat.
e/acc just says “accelerate AI”. Many VCs seem to think AI will remain a tool, they don’t want to be dethroned. Some e/acc are of course successionist, this is Musk and Beff with their talk about “all that matters is the light of intelligence is spread through the universe”.
Conflating all this is the fact that true successionism is way outside the Overton window and so people likely won’t be honest about their wishes. Larry Page famously espoused the view that it was “speciesist” for Musk to not want machines to replace humans. Most people find this position repugnant of course.
...What about some billionaire with any-promise-is-good-as-gold brain-implant tech convinces some powerful billionaire with immortal-power dreams that a merge with AI is the inevitable best way to bring about their big win?
Any half-done faked alignment partly inside some dry old dude's skull can then forever pretend the Panem-2100 hunterdrone games is what we desired all along!
Yeah; as expected, this is eugenics.
Any time you see some person or group saying that "only a select group of humans will/should survive", scratch the surface and you'll find that it's just a thin veneer over classic Josef Mengele-style eugenics.
1984. Brave New World. We.
