undefined | Better HN

Skip to content

Top Best Ask Show New Jobs

0 pointscjbprime4d ago0 comments

I've got access to a 192GB RAM Mac Studio, which is below the stated minimum RAM. Can swapping off fast disk be used to make it work out, especially since it's MoE?

0 comments

5 comments · 1 top-level

walrus014d ago· 4 in thread

Seems like a good way to shorten the lifespan of an NVME SSD significantly by using up its TB written lifespan, if you let it extensively swap. Also the performance will be absolutely abysmal like 0.1 tok/second.

cjbprimeOP3d ago

Both sentences are likely wrong. It's a written lifespan (technically an "erased" one), not a read lifespan. The weights are only needed from disk read-only. And Mac NVMe interfaces are surprisingly fast.

Edit: Oh, I think you maybe thought I meant swapping working RAM off disk? I didn't. I meant swapping weights off disk into RAM on-demand.

Yes, I meant that 192GB of RAM even with the worst quantization would result in such a large model going deep into disk swap when it entirely runs out of RAM. At least 100GB worth, if MacOS will even allow that without freezing or crashing or OOM killing the process.

cjbprimeOP3d ago

You still have a core misunderstanding. Only one layer of weights is required in memory at a time. A forward pass can be over-simplified as a matrix multiplication of each layer, one at a time.

There is no swapping of working RAM. We're just talking about loading the weights read-only data into RAM on-demand for each layer. It is only as slow as your storage interface.

smallerize4d ago

The LLM tools are smart enough to keep the weights on the disk and read-write stuff in RAM.

j / k navigate · click thread line to collapse