But why is that our default? I could just as well say we can't trust the estimates of p(survival), therefore we should fall back on a default action of not developing advanced AI.
Such progress has presented an existential threat only once previously in the past century, and that was the development of nuclear weapons. The jury is still out on whether that was the right decision. I don't think the argument that, "Well, nothing bad has happened yet!" is very persuasive in the face of possible extermination of the human race.
While there's plenty of positive effects of the march of technological progress, let's not pretend it's done us an unequivocally good turn. The state of the environment is enough to explain that.
Not going to happen. It's so potentially valuable economically and militarily that if one country doesn't develop it their rivals will.
If we outlaw AI, because of "risk", then people might experiment with it in secret, which has its own unique risks. If we study AI in the open, that has unique risks. If we descend further into authoritarianism to punish AI, that has risks. And no matter what we do about AI, the risks of nuclear war or life ending asteroid impacts remain.
> In philosophy, Pascal's mugging is a thought experiment demonstrating a problem in expected utility maximization. A rational agent should choose actions whose outcomes, when weighted by their probability, have higher utility. But some very unlikely outcomes may have very great utilities, and these utilities can grow faster than the probability diminishes. Hence the agent should focus more on vastly improbable cases with implausibly high rewards; this leads first to counter-intuitive choices, and then to incoherence as the utility of every choice becomes unbounded.
Curiously enough, this idea can be traced to one of the most prominent AI Safety advocates.
What are these "actual problems today"? More spam in Google results? Students cheating on assignments?
It seems clear that today's problems with LLMs are incredibly minor and unimportant compared to the potential extinction of humanity by future superintelligence.
I posted my specific proposal here: https://olegtrott.substack.com/p/recursion-in-ai-is-scary-bu...
Unlike previous ideas, it's implementable (once we have AGI-level language models) and works around the fact that much data on the Internet is false. I should probably call it Volition Extrapolated by Truthful Language Models.
That said, I see no reason to believe we are even on a path that can create AGI. LLMs don't actually understand or reason about anything.
Humans have innate desires that may conflict with the desires of other humans. A language model just looks for ways to continue texts. In doing so, it attempts to extrapolate what humans who authored the training data thought (If we ignore fabrications, which my approach proposes to address also)
> LLMs don't actually understand or reason about anything.
Current Transformer-based, SGD-trained language models fall short of AGI. But better algorithms can change that. And there is no reason to think that you'll see any warning signs in advance. No one said "I'll invent a faster Fourier transform algorithm in 5 years. Prepare yourselves."
I'm not sure about when the robots can build better robots and their own power sources - maybe worry about that when the time comes.
