undefined | Better HN

0 pointshimata411316d ago0 comments

inference compute is vastly different versus training, also it has to stay hot in vram which probably takes up most of it. There is limited use for THAT much compute as well, they are running things like claude code compiler and even then they're scratching the surface of the amount of compute they have.

Training currently requires nvidia's latest and greatest for the best models (they also use google TPU's now which are also technically the latest and greatest? However, they're more of a dual purpose than anything afaik so that would be a correct assesment in that case)

Inference can run on a hot potato if you really put your mind to it

0 comments

rafaelmn16d ago

I think I've heard multiple time that a large % of training compute for SoTA models is inference to generate training tokens, this is bound to happen with RL training

eru16d ago

They can run any number of inference experiments. Like a lot of the alignment work they have going on.

I am not saying this would be a great use of their compute, but idle is far from the only alternative. (Unless electricity is the binding constraint?)

himata4113OP16d ago

Electricity is charged whenever you use it or not, so very unlikely, but sure, they can find uses for it. Although they are not going to make that much money compared to claude code subscriptions.

eru16d ago

> Electricity is charged whenever you use it or not, [...]

Huh, what? You know you can turn off unused equipment, and at least my nvidia GPU can use more or less Watts even when turned on?

Or does Anthropic have a flatline deal for electricity and cooling?

1 more reply

j / k navigate · click thread line to collapse

0 pointshimata411316d ago0 comments

Inference can run on a hot potato if you really put your mind to it

0 comments

rafaelmn16d ago

I think I've heard multiple time that a large % of training compute for SoTA models is inference to generate training tokens, this is bound to happen with RL training

eru16d ago

They can run any number of inference experiments. Like a lot of the alignment work they have going on.

I am not saying this would be a great use of their compute, but idle is far from the only alternative. (Unless electricity is the binding constraint?)

himata4113OP16d ago

Electricity is charged whenever you use it or not, so very unlikely, but sure, they can find uses for it. Although they are not going to make that much money compared to claude code subscriptions.

eru16d ago

> Electricity is charged whenever you use it or not, [...]

Huh, what? You know you can turn off unused equipment, and at least my nvidia GPU can use more or less Watts even when turned on?

Or does Anthropic have a flatline deal for electricity and cooling?

1 more reply

j / k navigate · click thread line to collapse