undefined | Better HN

0 pointslolinder1y ago0 comments

Quantized to 4 bits you'll only need ~200GB! 5 4090s should cover it.

0 comments

You'll probably need 9 or more. 4090s have 24GB each.

Oops, I read 48 somewhere but that's wrong. Thanks.

htrp1y ago

a6ks however

I wonder if AutoAWQ works out of the box, given no architectural changes (?). That would be most straightforward together with vLLM for serving.

If an implementation had NVidia's Heterogeneous Memory Management implemented, then 192 GB RAM DDR5 + GPU VRAM would seem to be close.

Two 128gb Mac studios networked via thunderbolt 4?

This is actually a promising endeavor. Id love to see someone try that.

There's already at least one project that attempts this:

j / k navigate · click thread line to collapse