undefined | Better HN

0 pointsbijowo16761mo ago0 comments

the electricity was going to be consumed regardless whether you ask chatGPT or not.

It would have been either idle, or serving other users' requests.

so the incremental kWh consumption is zero, since costs are fixed and sunk.

as a rule of thumb you can lookup the power consumption of the latest nVidia chip, multiply by factor of two or three (to account for cpu/storage/cooling/network/infra)

0 comments

3 comments · 1 top-level

tredre31mo ago· 2 in thread

An idle GPU consumes almost nothing, a loaded (server-class) GPU can consume over 2kW.

Admittedly a single request isn't a full load, but claiming that a request makes no difference vs idle is misguided, in my opinion.

bijowo1676OP1mo ago

OpenAI GPU wont be idle for long because they have all other requests to serve. Over time there will be a certain % of idle GPUs, amortized across all hundreds of millions of requests they receive.

aoeusnth11mo ago

And idle% is causally connected to whether you make a request or not, surely? I don't understand how your mental model works.

j / k navigate · click thread line to collapse