undefined | Better HN

0 pointsreissbaker1y ago0 comments

16GB will be way too small unfortunately — this has over 3x the param count, so at best you're looking at a 24GB card with extreme 2bit quantization.

Really though if you're just looking to run models personally and not finetune (which requires monstrous amounts of VRAM), Macs are the way to go for this kind of mega model: Macs have unified memory between the GPU and CPU, and you can buy them with a lot of RAM. It'll be cheaper than trying to buy enough GPU VRAM. A Mac Studio with 192GB unified RAM is under $6k — two A6000s will run you over $9k and still only give you 96GB VRAM (and God help you if you try to build the equivalent system out of 4090s or A100s/H100s).

Or just rent the GPU time as needed from cloud providers like RunPod, although that may or may not be what you're looking for.

0 comments

timschmidt1y ago

Reasonably priced Epyc systems with up to 12 memory channels and support for several TB of system memory are now available. Used datacenter hardware is even less expensive. They are on par with the memory bandwidth available to any one of the CPU, GPU, or NPU in the highest end Macs, but capable of driving MUCH more memory. And much simpler to run Linux or Windows on.

reissbakerOP1y ago

I would be very curious to see pricing on Epyc systems with terabytes of RAM that cost less than $6k including the RAM...

timschmidt1y ago

Well the motherboard and CPU can be had for $1450. As they're built around standard cases and power supplies and storage, many folks like me will have those already - far less costly than buying the same from Apple if you don't. Spend what you want on ram, unlike with Apple, you can upgrade it any time.

Can't reuse my old parts on a brand new Mac, or upgrade it later if I find I need more. Lock-in is rough.

https://www.ebay.com/itm/315029731825?itmmeta=01HV561YV4AJG5...

1 more reply

hmottestad1y ago

Do you have any feel for the performance compared to the M3 Max?

Manabu-eo1y ago

LLM inference is mostly memory bound. An 12-channel Epyc Genoa with 4800MT/s DDR5 ram clocks at 460.8 GB/sec. It's more than the 400GB/s of the M3 Max, only part of that accessible for the CPU.

And on the Epyc System you can plug much more memory for when you need larger memory and PCI-E gpus, for when you need less faster memory.

Threadripper PRO is only 8-channel, but with memory overclocking it might reach numbers similar to those too.

3 more replies

dannyw1y ago

You can QLoRA decent models on 24GB VRAM. There’s also optimised kernels like Unsloth that are really VRAM efficient and good for hobbyists.

reissbakerOP1y ago

Yes, but I still don't think you'll be able to run Mixtral 8x22b with 16GB VRAM, or QLoRA it, even with Unsloth. It's much bigger than the original Mixtral.

j / k navigate · click thread line to collapse

0 pointsreissbaker1y ago0 comments

16GB will be way too small unfortunately — this has over 3x the param count, so at best you're looking at a 24GB card with extreme 2bit quantization.

Or just rent the GPU time as needed from cloud providers like RunPod, although that may or may not be what you're looking for.

0 comments

timschmidt1y ago

reissbakerOP1y ago

I would be very curious to see pricing on Epyc systems with terabytes of RAM that cost less than $6k including the RAM...

timschmidt1y ago

Can't reuse my old parts on a brand new Mac, or upgrade it later if I find I need more. Lock-in is rough.

https://www.ebay.com/itm/315029731825?itmmeta=01HV561YV4AJG5...

1 more reply

hmottestad1y ago

Do you have any feel for the performance compared to the M3 Max?

Manabu-eo1y ago

LLM inference is mostly memory bound. An 12-channel Epyc Genoa with 4800MT/s DDR5 ram clocks at 460.8 GB/sec. It's more than the 400GB/s of the M3 Max, only part of that accessible for the CPU.

And on the Epyc System you can plug much more memory for when you need larger memory and PCI-E gpus, for when you need less faster memory.

Threadripper PRO is only 8-channel, but with memory overclocking it might reach numbers similar to those too.

3 more replies

dannyw1y ago

You can QLoRA decent models on 24GB VRAM. There’s also optimised kernels like Unsloth that are really VRAM efficient and good for hobbyists.

reissbakerOP1y ago

Yes, but I still don't think you'll be able to run Mixtral 8x22b with 16GB VRAM, or QLoRA it, even with Unsloth. It's much bigger than the original Mixtral.

j / k navigate · click thread line to collapse