undefined | Better HN

0 pointsmanmal1mo ago0 comments

That’s what, 14GB/s? The GPU‘s VRAM can do 100x that.

0 comments

A discrete consumer GPU card doesn't have enough fast RAM to run a very large model that hasn't been quanitized to hell.

That's why all the projects streaming models into the GPU from an SSD popped up recently.

manmalOP1mo ago

Yes. There’s just no way to get above 1t/s that way with a large model.

j / k navigate · click thread line to collapse

A discrete consumer GPU card doesn't have enough fast RAM to run a very large model that hasn't been quanitized to hell.

That's why all the projects streaming models into the GPU from an SSD popped up recently.

manmalOP1mo ago

Yes. There’s just no way to get above 1t/s that way with a large model.

j / k navigate · click thread line to collapse