GPT inputs outgrow world chip and electricity capacity (opens in new tab)

(astralcodexten.com)

19 pointsquirkot2y ago14 comments

14 comments

Badly editorialized title.

Regarding the substance of the article, the curve from the 3 data points (1: 100x 2: 25x 3: 25x) could fit lots of different ways besides "growing 30x per generation".

danielscrubs2y ago

Really wish AMD could hurry up and implement a transparent module for gpu computing into llvm.

It’s not good that GPUs are this opaque.

dekhn2y ago

The are all the same arguments that wrongly predicted that DNA sequencing would overtake hard drive storage capacity.

People aren't going to do truly uneconomic things just to scale language models exponentially.

marsissippi2y ago

Interesting Pull quote:

GPT-4 needed about 50 gigawatt-hours of energy to train. Using our scaling factor of 30x, we expect GPT-5 to need 1,500, GPT-6 to need 45,000, and GPT-7 to need 1.3 million.

toomuchtodo2y ago

Assuming OpenAI is training exclusively using Microsoft compute, it should be low carbon energy.

https://news.microsoft.com/europe/features/as-the-world-goes...

Nullabillity2y ago

Until we're in a world of 100% renewables, anything you do that uses renewables just pushes someone else to use more fossil fuel instead. Access to renewables isn't a blanket excuse to waste it.

(Plus "renewable" is kind of a misnomer, there are still limited resources going into the production and maintenance of those power plants too.)

2 more replies

fulafel2y ago

Do the dynamics of PPAs and RECs really translate to lowering electricity production CO2 footprint? It seems that it would work only when there is significantly more demand for that kind of electricity that would otherwise we produced, but since wind and solar are competitive on their own, and in many places there's no carbon cap-and-trade system to prevent the increased power demand leaking over to fossil energy, it's not obvious.

1 more reply

krunck2y ago

There is finite low carbon energy available. Would they choose to slow the rate of training the next generations of LLMs and use only greener every sources or would they go full speed regardless of the source of the energy? I think they would do the later in order to be first in the market with a next generation LLM.

1 more reply

RecycledEle2y ago

New technologies experience very rapid exponential growth for a while.

This should not be a surprise.

j / k navigate · click thread line to collapse

14 comments

fulafel2y ago

Badly editorialized title.

Regarding the substance of the article, the curve from the 3 data points (1: 100x 2: 25x 3: 25x) could fit lots of different ways besides "growing 30x per generation".

danielscrubs2y ago

Really wish AMD could hurry up and implement a transparent module for gpu computing into llvm.

It’s not good that GPUs are this opaque.

dekhn2y ago

The are all the same arguments that wrongly predicted that DNA sequencing would overtake hard drive storage capacity.

People aren't going to do truly uneconomic things just to scale language models exponentially.

marsissippi2y ago

Interesting Pull quote:

GPT-4 needed about 50 gigawatt-hours of energy to train. Using our scaling factor of 30x, we expect GPT-5 to need 1,500, GPT-6 to need 45,000, and GPT-7 to need 1.3 million.

toomuchtodo2y ago

Assuming OpenAI is training exclusively using Microsoft compute, it should be low carbon energy.

https://news.microsoft.com/europe/features/as-the-world-goes...

Nullabillity2y ago

Until we're in a world of 100% renewables, anything you do that uses renewables just pushes someone else to use more fossil fuel instead. Access to renewables isn't a blanket excuse to waste it.

(Plus "renewable" is kind of a misnomer, there are still limited resources going into the production and maintenance of those power plants too.)

2 more replies

fulafel2y ago

1 more reply

krunck2y ago

1 more reply

RecycledEle2y ago

New technologies experience very rapid exponential growth for a while.

This should not be a surprise.

j / k navigate · click thread line to collapse