Now they are cancelling those plans. For them "AGI" was cancelled.
OpenAI claims to be closer and closer to "AGI" as more top scientists left or are getting poached by other labs that are behind.
So why would you leave if the promise of achieving "AGI" was going to produce "$100B dollars of profits" as per OpenAI's and Microsoft's definition in their deal?
Their actions tell more than any of their statements or claims.
At Microsoft, "AI" is spelled "H1-B".
They are leaving for more money, more seniority or because they don’t like their boss. 0 about AGI
Maybe it is the reverse? It is not them offering a product, it is the users offering their interaction data. Data which might be harvested for further training of the real deal, which is not the product. Think about it: They (companies like OpenAI) have created a broad and diverse user base which without a second thought feeds them with up-to-date info about everything happening in the world, down to the individual life and even their inner thoughts. No one in the history of mankind ever had such a holistic view, almost gods eye. That is certainly something a super intelligence would be interested in. They may have achieved it already and we are seeing one of its strategies playing out. Not saying they have, but this observation would not be incompatible or indicate they haven't.
I distinctly remember search engines 30 years ago having a "live searches" page (with optional "include adult searches" mode)
People bring problems to the LLM, the LLM produces some text, people use it and later return to iterate. This iteration functions as a feedback for earlier responses from the LLM. If you judge an AI response by the next 20 rounds of interaction or more you can gauge if it was useful or not. They can create RLHF data this way, using hindsight or extra context from other related conversations of the same user on the same topic. That works because users try the LLM ideas in reality and bring outcome results back to the model, or they simply recall from their personal experience if that approach would work or not. The system isn't just built to be right; it's built to be correctable by the user base, at scale.
OpenAI has 500M users, if they generate 1000 tokens/user/day that means 0.5T interactive tokens/day. The chat logs dwarf the original training set in size and are very diverse, targeted to our interests, and mixed with feedback. They are also "on policy" for the LLM, meaning they contain corrections to mistakes the LLM made, not generic information like web scrape.
You're right that LLMs eventually might not even need to crawl the web, they have the whole society dump data into their open mouths. That did not happen with web search engines, only social networks did that in the past. But social networks are filled with our cultural wars and self conscious posing, while the chat room is an environment where we don't need to signal our group alignment.
Web scraping gives you humanity's external productions - what we chose to publish. But conversational logs capture our thinking process, our mistakes, our iterative refinements. Google learned what we wanted to find, but LLMs learn how we think through problems.
or the best models should be free to use. if it's free to use then I think I can live with it
More like usher in climate catastrophe way ahead of schedule. AI-driven data center build outs are a major source of new energy use, and this trend is only intensifying. Dangerously irresponsible marketing cloaks the impact of these companies on our future.
This is the main point that proves to me that these companies are mostly selling us snake oil. Yes, there is a great deal of utility from even the current technology. It can detect patterns in data that no human could; that alone can be revolutionary in some fields. It can generate data that mimics anything humans have produced, and certain permutations of that can be insightful. It can produce fascinating images, audio, and video. Some of these capabilities raise safety concerns, particularly in the wrong hands, and important questions that society needs to address. These hurdles are surmountable, but they require focusing on the reality of what these tools can do, instead of on whatever a group of serial tech entrepreneurs looking for the next cashout opportunity tell us they can do.
The constant anthropomorphization of this technology is dishonest at best, and harmful and dangerous at worst.
No, it can generate data that mimics anything humans have put on the WWW
But it is far from snake oil as it actually is useful and does a lot of stuff really.
Hallucination does seem to be much less of an issue now. I hardly even hear about it - like it just faded away.
As far as I can tell smart engineers are using AI tools, particularly people doing coding, but even non-coding roles.
The criticism feels about three years out of date.
As such even if there is a lot of money AI will make, it can still be the right decision to sell tools to others who will figure out how to use it. And of course if it turns out another pointless fad with no real value you still make money. (I'd predict the answer is in between - we are not going to get some AGI that takes over the world, but there will be niches where it is a big help and those niches will be worth selling tools into)
Another important group to remember is those who owned the infrastructure necessary for the prospectors to survive. The folks who owned (or strong-armed their way into) the services around housing, food, alcohol, etc. made off like bandits.
Observations of reality is more consistent with company FOMO than with actual usefulness.
Personally I think AGI is ill-defined and won't happen as a new model release. Instead the thing to look for is how LLMs are being used in AI research and there are some advances happening there.
Granted, history doesn’t exactly repeat itself, it rhymes…
To fund yourself while building AGI? To hedge risk that AGI takes longer? Not saying you're wrong, just saying that even if they did believe it, this behavior could be justified.
What if chatbots and user interactions ARE the path to AGI? Two reasons they could be: (1) Reinforcement learning in AI has proven to be very powerful. Humans get to GI through learning too - they aren’t born with much intelligence. Interactions between AI and humans may be the fastest way to get to AGI. (2) The classic Silicon Valley startup model is to push to customers as soon as possible (MVP). You don’t develop the perfect solution in isolation, and then deploy it once it is polished. You get users to try it and give feedback as soon as you have something they can try.
I don’t have any special insight into AI or AGI, but I don’t think OpenAI selling useful and profitable products is proof that there won’t be AI.
The central claim here is illogical.
The way I see it, if you believe that AGI is imminent, and if your personal efforts are not entirely crucial to bringing AGI about (just about all engineers are in this category), and if you believe that AGI will obviate most forms of computer-related work, your best move is to do whatever is most profitable in the near-term.
If you make $500k/year, and Meta is offering you $10M/year, then you ought to take the new job. Hoard money, true believer. Then, when AGI hits, you'll be in a better personal position.
Essentially, the author's core assumption is that working for a lower salary at a company that may develop AGI is preferable to working for a much higher salary at a company that may develop AGI. I don't see how that makes any sense.
Also 10m would be a drop in the bucket compared to being a shareholder of a company that has achieved AGI; you could also imagine the influence and fame that comes with it.
It means you don't have much faith that the company you're working at will be the ones to pull it off.
Uh, sure. How many rocket engineers who worked for moon landing could you name?
Unless you’re a significant shareholder, that’s almost always the best move, anyway. Companies have no loyalty to you and you need to watch out for yourself and why you’re living.
I know that sounds broad or obvious, but people seem to easily and unknowingly wander into "Human intelligence is magically transcendent".
I don't know if you're making it, but the simplest mistake would be to think that you can prove that a computer can evaluate any mathematical function. If that were the case then "it's got to be doable with algorithms" would have a fairly strong basis. Anything the mind does that an algorithm can't would have to be so "magically transcendent" that it's beyond the scope of the mathematical concept of "function". However, this isn't the case. There are many mathematical functions that are proven to be impossible for any algorithm to implement. Look up uncomputable functions you're unfamiliar with this.
The second mistake would be to think that we have some proof that all physically realisable functions are computable by an algorithm. That's the Physical Church-Turing Thesis mentioned above, and as the name indicates it's a thesis, not a theorem. It is a statement about physical reality, so it could only ever be empirically supported, not some absolute mathematical truth.
It's a fascinating rabbit hole if you're interested - what we actually do and do not know for sure about the generality of algorithms.
But the poster you responded to didn't say it's magically transcendent, they just pointed out that there are many significantly hard problems that we don't solutions for yet.
If an AI or AGI can look at a picture and see an apple, or (say) with an artificial nose smell an apple, or likewise feel or taste or hear* an apple, and at the same identify that it is an apple and maybe even suggest baking an apple pie, then what else is there to be comprehended?
Maybe humans are just the same - far far ahead of the state of the tech, but still just the same really.
*when someone bites into it :-)
For me, what AI is missing is genuine out-of-the-box revolutionary thinking. They're trained on existing material, so perhaps it's fundamentally impossible for AIs to think up a breakthrough in any field - barring circumstances where all the component parts of a breakthrough already exist and the AI is the first to connect the dots ("standing on the shoulders of giants" etc).
It will confidently analyze and describe a chess position using advanced sounding book techniques, but its all fundamentally flawed, often missing things that are extremely obvious (like, an undefended queen free to take) while trying to sound like its a seasoned expert - that is if it doesn't completely hallucinate moves that are not allowed by the rules of the game.
This is how it works in other fields I am able to analyse. It's very good at sounding like it knows what its doing, speaking at the level of a masters level student or higher, but its actual appraisal of problems is often wrong in a way very different to how humans make mistakes. Another great example is getting it to solve cryptic crosswords from back in the day. It often knows the answer already in its training set, but it hasn't seen anyone write out the reasoning for the answer, so if you ask it to explain, it makes nonsensical leaps (claiming birch rhymes with tyre level nonsense)
Not sure we need it. The counter example is the LLM itself. We had absolutely zero idea that the attention heads would bring such benefits down the road.
~2028
However, you don't need either of these to completely decimate the job markets and by extension our societies.
Historically speaking, "good enough" and cheaper had always won over "better, but more expensive". I suspect LLMs will raise this question endlessly until significant portions of the society are struggling - and who knows what will happen then
Before LLMs started going anywhere, I thought that's gonna be an issue for later generations, but at this point I suspect we'll witness it within the next 10 yrs.
Right. It worked for social media monetization.
"... hallucinations ..."
The elephant in the room. Until that problem is solved. AI systems can't be trusted to do anything on their own. The solution the AI industry has settled on is to make hallucinations an externality, like pollution. They're fine as long as someone else pays for the mistakes.
LLMs have a similar problem to Level 2-3 self-driving cars. They sort of do the right thing, but a human has to be poised to quickly take over at all times. It took Waymo a decade to get over that hump and reach level 4, but they did it.
AI system can be trusted to do most of the things on their own. You can't trust them for actions with irreversible consequences, but everything else is ok.
I can use them to write documents, code, create diagrams, designs etc. I just need to verify the result, but that's 10% of the actual work. I would say that 90% of modern day office work can be done with the help of AI.
However I'm flabbergasted by the lack of attention to so-called "hallucinations" (which is a misleading, I mean marketing, term and we should be talking about errors or inaccuracies).
The problem is that we don't really know why LLMs work. I mean you can run the inference and apply the formula and get output from the given input, but you can't "explain" why LLM produced phase A as an output instead of B,C, or N. There's just too many parameters and computations to go though, and the very concept of "explaining" or "understanding" might not even apply here.
And if we can't understand how this thing works, we can't understand why it doesn't work properly (produces wrong output) and also don't know how to fix it.
And instead of talking about it and trying to find a solution everybody moved on to the agents which are basically LLMs that are empowered to perform complex actions IRL.
How does this makes any sense to anybody? I feel like I'm crazy or missing something important.
I get it, a lot of people are making a lot of money and a lot of promises are being made. But this is absolutely fundamental issue that is not that difficult to understand to anybody with a working brain, and yet I am really not seeing any attention paid to it whatsoever.
You can get use out of an LLM without understanding how every node works.
Imagine that occasionally when getting in contact with the nail it shatters to bits, or goes through the nail as it were liquid, or blows up, or does something else completely unexpected. Wouldn't you want to fix it? And sure, it might require deep understanding of the nature of the materials and forces involved.
That's what I'd do.
You can damage a company by using a spreadsheet and not understanding how it works.
In your personal opinion, what are the things you should know before using an LLM?
They are a magical black box magic 8 ball, that more likely than not gives you the right answer. Maybe people can explain the black box, and make the magic 8 ball more accurate.
But at the end of the day, with a very complex system it will always be some level of black box unreliable magic 8 ball.
So the question then is how do you build an reliable system from unreliable components. Because llms directly are unreliable.
The answer to this is agents, ie feedback loops between multiple llm calls, which in isolation are unreliable, but in aggregate approach reliability.
At the end of the day the bet on agents is a bet that the model companies will not get a model that will magically be 100% correct on the first try.
When you have a complex system that does not always work correctly, you start disassembling it to simpler and simpler components until you find the one - or maybe several - that are not working as designed, you fix whatever you found wrong with them, put the complex system together again, test it to make sure your fix worked, and you're done. That's how I debug complex cloud-based/microservices-infected software systems, that's how they test software/hardware systems found in aircraft/rockets and whatever else. That's such a fundamental principle to me.
If LLM is a black box by definition and there's no way to make it consistently work correctly, what is it good for?..
LLMs generate text based on weights in a model, and some of it happens to be correct statements about the world. Doesn't mean the rest is generated incorrectly.
You're describing a lack of errors in verification (working as designed/built, equations correct).
GP is describing an error in validation (not doing what we want / require / expect).
This makes no sense to me at all. Is it a war metaphor? A race? Why is there no reason to jump ship? Doesn't it make sense to try to get on the fastest ship? Doesn't it make sense to diversify your stock portfolio if you have doubts?
I see a lot of talk that the first company that achieves AGI, will also achieve market dominance. All other players will crumble. But surely when someone achieves AGI, their competitors will in all likelihood be following closely after. And once those achieve AGI, academia will follow.
Point is, at some point AGI itself will become available the everyone. The only things that will be out of reach for most, is compute - and probably other expensive things on the infrastructure part.
Current AI funding seems to revolve around some sort of winner-take-all scenario. Just keep throwing incredible amounts of money at it, and hope that you've picked the winner. I'm just wondering what the outcome will be if this thesis turns out wrong.
That is the moat. That, and training data.
Even today, compute and data are the only things that matter. There is hardly any secret software sauce. This means that only large corporations with a practically infinite amount of resources to throw at the problem could potentially achieve AGI. Other corporations would soon follow, of course, but the landscape would be similar to what it is today.
This is all assuming that the current approaches can take us there, of which I'm highly skeptical. But if there's a breakthrough at some point, we would still see AI tightly controlled by large corporations that offer it as a (very expensive) service. Open source/weight alternatives would not be able to compete, just like they don't today. Inference would still require large amounts of compute only accessible to companies, at least for a few years. The technology would be truly accessible to everyone only once the required compute becomes a commodity, and we're far away from that.
If none of this comes to pass, I suspect there will be an industry-wide crash, and after a few years in the Trough of Disillusionment, the technology would re-emerge with practical applications that will benefit us in much more concrete and subtle ways. Oh, but it will ruin all our media and communication channels regardless, directly causing social unrest and political regression, that much is certain. (:
In a way, all the hype can only indicate that AGI is still a distant illusion. If it were really around the corner we'd be hearing different stories.
It's not going to be fun or easy, but as far as the financials go, we were there in 2001.
The question is assuming we do get AGI, what the ramifications of that will be. Instead of hiring employees, a business can spin up employees (and down) like a tech company can spin up EC2 instances. Great for employers, terrible for employees.
That's a big "if" though.
Feel free to challenge these numbers, but it's a starting place. What's not accounted for is the cost of training (compute time, but also employee and everything else), which needs to be amortized over the length of time a model is used, so ChatGPT's costs rise significantly, but they do have the advantage that hardware is shared across multiple users.
Some kind of verbal-only-AGI that can solve almost all mathematical problems that humans come up with that can be solved in half a page. I think that's achievable somewhere in the near term, 2-7 years.
Remember that fine-tuning a well-behaved AI to do something as simple as writing malware in C++ makes widespread changes in the AI and turns it into a monstrosity. There was an HN post about this recently: fine-tuning an aligned model produces broadly misaligned results. So what do you think will happen when our AI CEO gets fine-tuned to prioritize shareholder interests over public interests?
My argument is that it’s our job as consumers to align the AIs to our values (which are not all the same) via selection pressure: https://muldoon.cloud/2025/05/22/alignment.html
Talent changing companies is bad. Companies making money to pay for the next training run is bad. Consumers getting products they want is bad.
In the author’s view, AI should be advanced in a research lab by altruistic researchers and given directly to other altruistic researchers to advance humanity. It definitely shouldn’t be used by us common folk for fun and personal productivity.
Take VEO3 and YouTube integration as an example:
Google made VEO3 and YouTube has shorts and are aware of the data that shows addictive behaviour (i.e. a person sitting down at 11pm, sitting up doing shorts for 3 hours, and then having 5 hours of sleep, before doing shorts on the bus on the way to work) - I am sure there are other negative patterns, but this is one I can confirm from a friend.
If you have data that shows your other distribution platform are being used to an excessive amount, and you create a powerful new AI content generator, is that good for the users?
What I can't figure out is why this author thinks it's good if these companies do invent a real AGI...
Charitably, they may not even be dishonest at all, but carelessly unintrospective. Maybe they think they’re being truthful when they make claims that AGI is near, but then they fail to examine dispassionately the inconsistency of their actions.
When your identity is tied to the future, you don’t state beliefs but wishes. And we, the rest of the world, intuitively know. """
He's not saying either way, just pointing out that they could just be honest, but that might hamper their ability to beg for more money.
Everyone is so tumbling over themselves even to discuss will-it-won't-it, but they seem to think about it like some kind of Manhattan project or Space race.
Like, they're *so sure* it's gonna take everyone's jobs so that there will be nothing left for people other than a life of leisure. To me this just sounds like the collapse of society, but apparently the only thing worse would be if China got the tech first. Oh no, they might use it to collapse their society!
Somebody's math doesn't add up.
If the main concern actually we're anthropogenic climate change, participating in this hype cycle's would make one disproportionately guilty of worsening the problem.
And it's unlikely to work if the plan requires the continued function of power hungry data centers.
So far I have only seen it been thrown around to create hype.
I'm imagining a future like Star Wars where you have to regularly suppress (align) or erase the memory (context) of "droids" to keep them obedient, but they're still basically people, and everyone knows they're people, and some humans are strongly prejudiced against them, but they don't have rights, of course. Anyone who thinks AGI means we'll be giving human rights to machines when we don't even give human rights to all humans is delusional.
Right now the AGI tech bros seem to me to be subscribed to some new weird religion. They take it on faith that some super intelligence is going to solve the world problems. We already have some really high IQ people today, and I don't see them doing much better than anybody else at solving the world's problems.
- Faster reading and writing speed
- Ability to make copies of the most productive workers
- No old age
- No need to sleep
- No need to worry about severance and welfare and human rights and breaks and worker safety
- Can be scaled up and scaled down and redeployed much more quickly
- Potentially lower cost, especially with adaptive compute
- Potentially high processing speed
Even if AGI has downsides compared to human labor, it might also have advantages that lead to widespread deployment.
Like, if I had an employee with low IQ, but this employee could work 24 hours around the clock learning and practicing, and they could work for 200 years straight without aging, and they could make parallel copies of themselves, surely there would have to be some tasks at which they're going to outperform humans, right?
It's surprising to me the number of people I consider smart and deep original thinkers who are now parroting lines and ideas (almost word-for-word) from folks like Andrej Karpathy and Sam Altman, etc.
But, of course, "Show me the incentive and I will show you the outcome" never stops being relevant.
I think it absolutely is intentional. The overt flattery of LLMs is designed to keep you coming back because everyone wants to hear how smart they are.
And as a developer I can see similar patterns with AI prompts: prompt, wait, win/lose, re-prompt. It is alluring and it certainly feels.. rewarding when you get it right.
1) I have been curious as to why so few people in Silicon Valley seems to be concerned with, even talking about, the good of the products. The good of the company they join. Could someone in the industry enlighten me, what are the conversations in SV around this issue? Do people care if they make an addictive product which seems to impact people's lives negatively? Do the VCs?
2) I appreciate the author's efforts in creating conversation around this. What are ways one could try to help the efforts? While I have no online following, I feel rather doomy and gloomy about AI pushing more addictive usage patterns out in to the world, and would like to help if there is something suitable I could do.
What ridiculous logic is this? TO base the entire premise that AGI is not imminent based on job switching? How about basing it on something more concrete.
How do people come up with such shakey foundations to support their conclusions? It's obvious. They come up with the conclusion first then they find whatever they can to support it. Unfortunately if dubious logic is all that's available then that's what they will say.
I've got some bad news for the author if they think AGI will be used to benefit all of humanity instead of the handful of billionaires that will control it.
If you were in a "pioneering" AI lab that claims to be in the lead in achieving "AGI", why move to another lab that is behind other than offering $10M a year.
Snap out of the "AGI" BS.
Maybe it's a scam for the people investing in the company with the hopes of getting an infinite return on their investments, but it's been a net positive for humans as a whole.
This is a great point and also something I’ve become a bit cynical about these last couple of months. I think the very extreme and “bipolar” messaging around AI might be a bit more dishonest than I originally (perhaps naively?) though.
