Is there a law of thermodynamics which prevents AI from writing code which would train a better AI? Never learned that one in school.
And FYI here's OpenAI plan to align superintelligence: "Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence."
I guess people working there believe in magic.
> and you can wish an AI into existence.
Eh? People believe that self-improvement might happen when AI is around human-level.
You need to apply Wittgenstein here.
This appears to be true because you haven't defined "better". If you define it, it'll become obvious that this is either false or true, but if it is true it'll be obvious in a way that doesn't make it sound interesting anymore.
(For one thing our current "AI" don't come from "writing code", they just come from training bigger models on the same data. For another, making changes to code doesn't make it exponentially better, and instead breaks it if you're not careful.)
> I guess people working there believe in magic.
Yes, OpenAI was literally founded by a computer worshipping religious cult.
> People believe that self-improvement might happen when AI is around human-level.
Humans don't have a "recursive self-improvement" ability.
Also not obvious that an AI that was both "aligned" and "capable of recursive self-improvement" would choose to do it; if you're an AI and you're making a new improved AI, how do you know it's aligned? It sounds unsafe.
They do.
Humans can learn from new information, but also by iteratively distilling existing information or continuously optimizing performance on an existing task.
Mathematics is a pure instance of this, in the sense that all the patterns for conjectures and proven theorems are available to any entity to explore, no connection to the world needed.
But any information being analyzed for underlying patterns, or task being optimized for better performance, creates a recursive learning driver.
Finally, any time two or more humans compete at anything, they drive each other to learn and perform better. Models can do that too.
Are you arguing that all AI models are using the same network structure?
This is only true in the most narrow sense, looking at models that are strictly improvements over previous generation models. It ignores the entire field of research that works by developing new models with new structures, or combining ideas from multiple previous works.
The exception is when you care about efficiency (in training or inference costs) but at the limit or if you care about "better" then you don't.
Ok. So then I guess it isn't "just a belief that magic".
Instead, it is so true and possible that you think it is actually obvious!
I'm glad you got convinced in a singular post that recursive self improvement, in the obvious way, is so true and real that it is obviously true and not magic.
Better intelligence can be defined quite easily: something which is better at (1) modeling the world; (2) optimizing (i.e. solving problems).
But if that would be too general we can assume that general reasoning capability would be a good proxy for that. And "better at reasoning" is rather easy to define. Beyond general reasoning better AI might have access to wider range of specialized modeling tools, e.g. chemical, mechanical, biological modeling, etc.
> if it is true it'll be obvious in a way that doesn't make it sound interesting anymore.
Not sure what you mean. AI which is better at reasoning is definitely interesting, but also scary.
> they just come from training bigger models on the same data.
I don't think so. OpenAI refuses to tell us how they made GPT-4. I think a big part of it was preparing better, cleaner data sets. Google tells us that specifically improved Gemini's reasoning using specialized reasoning datasets. More specialized AI like AlphaGeometry use synthetic datasets.
> Yes, OpenAI was literally founded by a computer worshipping religious cult.
Practice is the sole criterion for testing the truth. If their beliefs led them to better practice then they are closer to truth than whatever shit you believe in. Also I see no evidence of OpenAI "worshipping" anything religion-like. Many people working there are just excited about possibilities.
> Humans don't have a "recursive self-improvement" ability.
Human recursive self-improvement is very slow because we cannot modify our brains' at will. Also spawning more humans takes time. And yet humans made huge amount of progress in the last 3000 years or so.
Imagine that instead of making a new adult human in 20 years you could make one in 1 minute with full control over neural structures, connections to external tools via neural links, precisely controlled knowledge & skills, etc.
>Yes, OpenAI was literally founded by a computer worshipping religious cult.
What cult is this?
I've been thinking about this recently. Personally, I've yet to see any compelling evidence that an LLM, let alone any AI, can operate really well "out of distribution". It's capabilities (in my experience) seem to be spanned by the data it's trained on. Hence, this supposed property that it can "train itself", generating new knowledge in the process, is yet to be proven in my mind.
That raises the question for me: why do OpenAI staff believe what they believe?
If I'm being optimistic, I suppose they may have seen unreleased tech, motivating their beliefs that seemingly AGI is on the horizon.
If I'm being cynical, the promise of AGI probably draws in much more investment. Thus, anyone with a stake in OpenAI has an incentive to promote this narrative of imminent AGI, regardless of how realistic it is technically.
This is of course just based on what I've seen and read, I'd love to see evidence that counter my claims.
I think the concern about out-of-distribution is overstated. If we train it on predicting machine learning papers, writing machine learning papers is not out-of-distribution.
You might say "but writing NOVEL papers" would be OOD; but there's no sharp boundary between old and new. Model's behavior is usually smooth, so it's not like it will output random bs if you try to predict 2025 papers. And predicting 2025 papers in 2024 all we need to do "recursive self-improvement". (There are also many ways to shift distribution towards where you want it to be, e.g. aesthetics tuning, guidance in diffusion models, etc. Midjourney does not faithfully replicate distribution in the input training set, it's specifically tuned to create more pleasing outputs. So I don't see "oh but we don't have 2025 papers in the training set yet!" being an insurmountable problem.)
But more generally, seeing models as interpolators is useful only to some extent. We use statistical language when training the models, but that doesn't mean that all output should be interpreted as statistics. E.g. suppose I trained a model which generates a plausible proofs. I can combine it with proof-checker (which is much easier than generating a proof), and wrap it into a single function `generate_proof` which is guaranteed to generate a correct proof (it will loop until a plausible proof checks out). Now the statistics do not matter much. It's just a function.
If there's such a thing as a general reasoning step, then all we need is a function which perform that. Then we just add an outer loop to explore a tree of possibilities using these steps. And further improvements might be in making these steps faster and better.
Does reasoning generalize? I'd say everything points to "yes". Math is used in variety of fields. We are yet to find something where math doesn't work. If you get somebody educated in mathematical modeling and give them a new field to model, they won't complain about math being out-of-distribution.
If you look at LLMs today, they struggle with outputting JSON. It's clearly not an out-of-distribution problem, it's a problem with training - the dataset was too noisy, it had too many examples where somebody requests a JSON but gets a JSON-wrapped-in-Markdown. It's just an annoying data cleanup problem, nothing fundamental. I think it's reasonable to assume that within 5 years OpenAI, Google, etc, will manage to clean up their datasets and train more capable, reliable models which demonstrate good reasoning capabilities.
FWIW I believe that if we hit a wall on a road towards AGI that might actually be good to buy more time to research what we actually want out of AGI. But I doubt that any wall will last more than 5 years, as it already seems almost within the reach...
I can see how such a pipeline can exist. I can imagine the problematic bit being the "validation system". In closed systems like mathematics, the proof can be checked with our current understanding of mathematics. However, I wonder if all systems have such a property. If, in some sense, you need to know the underlying distribution to check that a new data point is in said distribution, the system described above cannot find new knowledge without already knowing everything.
Moreover, if we did have such a perfect "validation system", I suppose the only thing the ML models are buying us is a more effective search of candidates, right? (e.g., we could also just brute force such a "validation system" to find new results).
Feel free to ignore my navel-gazing; it's fascinating to discuss these things.
Are you gonna to take a bet "AI won't be able to do X in 10 years" for some X which people can learn to do now? If you're unwilling to bet then you believe that AI would plausibly be able to perform any human job, including job of AI researcher.
Saying "well that is not physically impermissible" doesn't make it real.
In any case nobody has ever shown that recursive self-improvement "takes off", and nor is that what we should expect a priori.
