undefined | Better HN

0 pointsPragmaticPulp2y ago0 comments

Local LLMs do best on big GPUs, which aren’t an option on Macs.

What Apple has done with the M-series silicon is really impressive, but it’s not as fast as big, dedicated GPU silicon.

> like if the most cost-effective work boot of 2025 is produced by Gucci.

Apple M-silicon laptops are high quality, but they’re not the same as overpriced luxury goods. The price of entry level M1/M2 Macs is extremely reasonable, IMO.

0 comments

daniel-thompson2y ago

> Local LLMs do best on big GPUs, which aren’t an option on Macs.

For processing speed, Ms are fast but I agree not anywhere near a top Nvdia chip.

For memory size, the memory on an M chip can be used as graphics memory, so a person could get an M2 today with 128GB of graphics memory for ~ $5k. Not bad considering an Nvidia chip approaching that memory size is several times that much.

1 more reply

Maursault2y ago

> Local LLMs do best on big GPUs, which aren’t an option on Macs. What Apple has done with the M-series silicon is really impressive, but it’s not as fast as big, dedicated GPU silicon.

It'd be pretty silly for Apple to cram in big, dedicated GPU when 99.9% of their customers don't care and don't want to pay for it, especially considering that anyone that does want big, dedicated GPU can outboard as much GPU as they want.[1] And many seem to think that the onboard GPU along with Neural Engine should be adequate for local LLM.

[1] https://support.apple.com/en-us/HT208544

thoughtsimple2y ago

Only on Intel Macs which are going away soon (only 2019 Mac Pro is remaining outside of refurbished.)

Apple Silicon doesn't currently have any provision for external graphics cards.

Maursault2y ago

There are two issues.

     1) Apple Silicon won't support PCIe. 

     2) Apple doesn't want it to.

#2 means Apple is taking nVidia and AMD head on in the GPU space. Apple wants to control everything, and allowing these competitors on their platform is giving away too much. Because Apple Silicon scales better than competitors' hw, the desire for third party GPU is probably going to evaporate within a few generations of Apple Silicon. I mean, we'll see, but that is my best guess, because it seems like that was an intentional decision rather than oversight.

2 more replies

dacryn2y ago

they share their main memory with the GPU.

There is no cheaper way to have 50gb of memory allocated to your GPU compute pipeline (buy a 64gb macbook pro or studio). So yes big, only the fast aspect remains a reason to buy expensive dedicated gpu's

samwillis2y ago

> it’s not as fast as big, dedicated GPU silicon

It's not yet. Apples combined memory architecture enables them to significantly cut the cost of bundling more memory for various GPU, Neural or other domain specific cores. I believe they will catch up or even overtake Nvidia as the leading AI platform.

Desktop AI class GPUs are hell expensive. If Apple can get something 50% as performant, but in an iPhone Pro or MacBook Air, thats going to change so very much.

xipix2y ago

It's so funny to see Apple's "combined memory architecture" praised. When Intel did this a decade+ ago with its integrated GPUs, it was roundly slated.

comex2y ago

Well, the difference is that compared to a baseline where GPUs typically get faster RAM than CPUs, Apple is providing fast RAM to both, whereas Intel provided slow RAM to both. Of course, that’s hardly Intel’s fault given that their integrated GPUs are aimed at the low end and are usually paired with external socketed RAM.

gpderetta2y ago

but AI class GPUs are expensive because NVidia can pretty much ask whatever it wants as it has no competition. If Apple were to become competitive in the area, things can change.

diffeomorphism2y ago

> The price of entry level M1/M2 Macs is extremely reasonable, IMO.

Caveats: In the US, with low ram and low storage.

Without these caveats you look at about 2000€ or so (m2 macbook air with 16gb of ram and non-halved storage performance in Germany). That can still be a decent deal for what you get, but "extremely reasonable" -- not quite.

j / k navigate · click thread line to collapse

0 comments

daniel-thompson2y ago

> Local LLMs do best on big GPUs, which aren’t an option on Macs.

For processing speed, Ms are fast but I agree not anywhere near a top Nvdia chip.

1 more reply

Maursault2y ago

> Local LLMs do best on big GPUs, which aren’t an option on Macs. What Apple has done with the M-series silicon is really impressive, but it’s not as fast as big, dedicated GPU silicon.

[1] https://support.apple.com/en-us/HT208544

thoughtsimple2y ago

Only on Intel Macs which are going away soon (only 2019 Mac Pro is remaining outside of refurbished.)

Apple Silicon doesn't currently have any provision for external graphics cards.

Maursault2y ago

There are two issues.

     1) Apple Silicon won't support PCIe. 

     2) Apple doesn't want it to.

2 more replies

dacryn2y ago

they share their main memory with the GPU.

samwillis2y ago

> it’s not as fast as big, dedicated GPU silicon

Desktop AI class GPUs are hell expensive. If Apple can get something 50% as performant, but in an iPhone Pro or MacBook Air, thats going to change so very much.

xipix2y ago

It's so funny to see Apple's "combined memory architecture" praised. When Intel did this a decade+ ago with its integrated GPUs, it was roundly slated.

comex2y ago

gpderetta2y ago

but AI class GPUs are expensive because NVidia can pretty much ask whatever it wants as it has no competition. If Apple were to become competitive in the area, things can change.

diffeomorphism2y ago

> The price of entry level M1/M2 Macs is extremely reasonable, IMO.

Caveats: In the US, with low ram and low storage.

j / k navigate · click thread line to collapse