Idk. Folks much smarter than I seem worried so maybe I should be too but it just seems like such a long shot.
The vast majority of datacenters currently in production will be entirely powered by carbon free energy. From best to worst:
1. Meta: 100% renewable
2. AWS: 90% renewable
3. Google: 64% renewable with 100% renewable energy credit matching
4. Azure: 100% carbon neutral
[1]: https://sustainability.fb.com/energy/
[2]: https://sustainability.aboutamazon.com/products-services/the...
[3]: https://sustainability.google/progress/energy/
[4]: https://azure.microsoft.com/en-us/explore/global-infrastruct...
If imaginary cloud provider "ZFQ" uses 10MW of electricity on a grid and pays for it to magically come from green generation, that means 10MW of other loads on the grid were not powered by green energy, or 10MW of non-green power sources likely could have been throttled down/shut down.
There is no free lunch here; "we buy our electricity from green sources" is greenwashing bullshit.
Even if they install solar on the roofs and wind turbines nearby - that's still electrical generation capacity that could have been used for existing loads. By buying so many solar panels in such quantities, they affect availability and pricing of all those components.
The US, for example, has about 5GW of solar manufacturing capacity per year. NVIDIA sold half a million H100 chips in one quarter, each of which uses ~350W, which means in a year they're selling enough chips to use 700MW of power. That does not include power conversion losses, distribution, cooling, and the power usage of the host systems, storage, networking, etc.
And that doesn't even get into the water usage and carbon impact of manufacturing those chips; the IC industry uses a massive amount of water and generates a substantial amount of toxic waste.
It's hilarious how HN will wring its hands over how much rare earth metals a Prius has and shipping it to the US from Japan, but ask about the environmental impacts of AI and it's all "pshhtt, whatever".
What are your timelines here? "Catastrophic" is vague but I'd put the climate change meaningfully affecting the quality of life of average westerner at end of century, while AGI could be before the middle of the century.
We have surpassed the 1.5°C goal and are on track towards 3.5°C to 5°C. This accelerates the climate change timeline so that we'll see effects postulated for the end of the century in about ~20 years.
IMO, we should pause this for now and put these resources (human and capital) towards reducing the impact of global warming.
So yes, the insiders very likely know a thing or two that the rest of us don’t.
The most obvious reason is costs - if it costs many millions to train foundation models, they don't have a ton of experiments sitting around on a shelf waiting to be used. They may only get 1 shot at the base-model training. Sure productization isn't instant, but no one is throwing out that investment or delaying it longer than necessary. I cannot fathom that you can train an LLM at like 1% size/tokens/parameters to experiment on hyper parameters, architecture, etc and have a strong idea on end-performance or marketability.
Additionally, I've been part of many product launches - both hyped up big-news-events and unheard of flops. Every time, I'd say that 25-50% of the product is built/polished in the mad rush between press event and launch day. For an ML Model, this might be different, but again see above point.
Sure products may be planned month/years out, but OpenAI didn't even know LLMs were going to be this big a deal in May 2022. They had GPT-2 and GPT-3 and thought they were fun toys at that time, and had an idea for a cool tech demo. I think that OpenAI (and Google, etc) are entirely living day-to-day with this tech like those of us on the outside.
I agree, and they are also living in a group-think bubble of AI/AGI hype. I don't think you'd be too welcome at OpenAI as a developer if you didn't believe they are on the path to AGI.
What we're going to see over next year seems mostly pretty obvious - a lot of productization (tool use, history, etc), and a lot of efforts with multimodality, synthetic data, and post-training to add knowledge, reduce brittleness, and increase benchmark scores. None of which will do much to advance core intelligence.
The major short-term unknown seems to be how these companies will be attempting to improve planning/reasoning, and how successful that will be. OpenAI's Schulman just talked about post-training RL over longer (multi-reasoning steps) time horizons, and another approach is external tree-of-thoughts type scaffolding. These both seem more about maximizing what you can get out of the base model rather than fundamentally extending it's capabilities.
If you've been working on AI, you've seen everything go up and to the right for a while - who really benefits from pointing out that a slowdown is occurring? Who is incentivized to talk about how the benefits from scaling are slowing down or the publicly available internet-scale corpuses are running out? Not anyone who trains models and needs compute, I can tell you that much. And not anyone who has a financial interest in these companies either.
That's easy, we just need to make meatspace people stupider. Seems to be working great so far.
"Meanwhile what they have created is just a very impressive hot water bottle that turns a crank."
"Meanwhile what they have created is just a very impressive rock where neutrons hit other neutrons."
The point isn't how it works, the point is what it does.
Honestly? I'm not too worried
We've seen how the google employee that was "seeing a conscience" (in what was basically GPT-2 lol) was a nothing burger
We've seen other people in "AI Safety" overplay their importance and hype their CV more than actually do any relevant work. (Usually also playing the diversity card)
So, no, AI safety is important but I see it attracting the least helpful and resourceful people to the area.
That has proven to be a mistake
Given the model is probabilistic and does many things in parallel, its output can be understood as a mixture, e.g. 30% trash, 60% rehashed training material, 10% reasoning.
People probe model in different ways, they see different results, and they make different conclusions.
E.g. somebody who assumes AI should have impeccable logic will find "trash" content (e.g. incorrectly retrieved memory) and will declare that the whole AI thing is overhyped bullshit.
Other people might call model a "stochastic parrot" as they recognize it basically just interpolates between parts of the training material.
Finally, people who want to probe reasoning capabilities might find it among the trash. E.g. people found that LLMs can evaluate non-trivial Python code as long as it sends intermediate results to output: https://x.com/GrantSlatton/status/1600388425651453953
I interpret "feel the AGI" (Ilya Sutskever slogan, now repeated by Jan Leike) as a focus on these capabilities, rather than on mistakes it makes. E.g. if we go from 0.1% reasoning to 1% reasoning it's a 10x gain in capabilities, while to an outsider it might look like "it's 99% trash".
In any case, I'd rather trust intuition of people like Ilya Sutskever and Jan Leike. They aren't trying to sell something, and overhyping the tech is not in their interest.
Regarding "missing something really critical", it's obvious that human learning is much more efficient than NN learning. So there's some algorithm people are missing. But is it really required for AGI?
And regarding "It cannot reason" - I've seen LLMs doing rather complex stuff which is almost certainly not in the training set, what is it if not reasoning? It's hard to take "it cannot reason" seriously from people
Nobody defines what they’re trying to do as “useful AI” since that’s a much more weasily target, isn’t it?
The whole industry at this point is acting like the tobacco industry back when they first started getting in hot water. No doubt the prophecies about imminent AGI will one day look to our descendents exactly like filters on cigarettes. A weak attempt to prevent imminent regulation and reduced profitability as governments force an out of control industry to deal with the externalities involved in the creation of their products.
If it wasn't abundantly clear...I agree with you that AGI is infinitely far away. Its the damage that's going to be caused by sociopaths (Sam Altman at the top of the list) in attempting to justify the real things they want (money) in their march towards that impossible goal that concerns me.
