undefined | Better HN

0 pointsliuliu5y ago0 comments

They offload parameters, gradients and optimizer states (such as moment, velocity and exponential avg of these in Adam) into CPU memory.

0 comments

1 comments · 1 top-level

p1esk5y ago

They did all that before: https://arxiv.org/abs/2101.06840, but they could only fit a model with 13B weights on a single V100.

j / k navigate · click thread line to collapse