Limitations of self-improvement: The author uses the example of a self-improving computer program, CompilerOne, which can optimize its own source code to create a more efficient version, CompilerTwo. However, when CompilerTwo is fed the same source code, it can only replicate itself, not create a more advanced CompilerThree. This illustrates the inherent limitations of self-improvement in machines.
The role of human civilization: The author suggests that recursive self-improvement is more applicable to human civilization as a whole, rather than individual intelligence. Over time, humans have developed a wide range of cognitive and physical tools, which have enabled us to invent even more powerful tools. This collective progress is different from an individual machine's ability to improve itself.
The importance of collaboration: The author argues that major breakthroughs often come from many people drawing inspiration from one another, rather than a single individual or machine working in isolation. This suggests that a single AI, even if it could improve itself, would not be as effective as a collaborative network of individuals.
The challenge of optimizing for generality: The author questions the extent to which a system can be optimized for every possible situation, including those it has never encountered before. While some improvement may be possible, the author argues that the idea of an intelligence explosion implies virtually limitless optimization, which is a strong and potentially unfounded claim.
The role of human decision-making: The author concludes that for the foreseeable future, technological progress will be driven by humans using previously invented tools to invent new ones. The fate of our species will depend on human decision-making, rather than the actions of a superhumanly intelligent AI.
On limitations of self-improvement: The very nature of AI is self-improvement. AI becomes smarter with more data. Thus, the only thing it needs is more data.
On the role of human civilizations: If you give an AI a goal, it will try to achieve that goal, including trying to become smarter in order to achieve that goal. Humans also have goals - to survive and reproduce. Furthermore, I constantly see the argument that AIs don't manifest itself in physical form as a reason it can't improve itself. This seems extremely short sighted. We're about put AI into every single thing. We're going to have AI robots, AI dogs, AI cleaners, AI toasters. The physical form is not a long-term barrier for an AI to self-improve.
On the importance of collaboration: I don't see how this prevents an AI from improving itself. Let's say an AI model is as intelligent as Albert Einstein. This AI can simply spawn many copies of itself. The AI will allocate 1 million Einsteins to study different areas of physics. 1 million Einsteins for chemistry. 1 million Einsteins to learn human persuasion. Etc. Now they can all come together and share ideas.
On the challenge of generality: Doesn't ChatGPT prove that an AI can have general intelligence?
On the role of human decision-making: Sure, the foreseeable future, tech will still be driven by humans. But the author argues that the singularity will never happen and computers won't make themselves smarter. It sounds like the author actually convinced himself that machines could make themselves smarter, just not yet?
And finally, on this quote from the author:
>We’re a long way off from being able to create a single human-equivalent A.I., let alone billions of them.
If we can create one, creating a billion will be the easy part.
I’m not entirely sure what the authors argument is.
If an intelligence explosion occurs with humans and AI working in collaboration, it is still an intelligence explosion. No?
Computers will make themselves smarter - with the help of humans. They already are. RLHF is a perfect example of this.
They’re arguing that you can never take the H out of RLHF? I don’t see why not. We are physical systems just like AI. Our general search algorithm is to find systems that maximize accurate predictions. Minimize surprise. This can be coded.
