PyTorch Library for Running LLM on Intel CPU and GPU (opens in new tab)

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... )

This is a huge problem because in theory the Arc A770 is faster! It's theoretical performance (TFLOPS) is more than twice as fast as an Nvidia 4060 (see: https://cdn.mos.cms.futurecdn.net/Q7WgNxqfgyjCJ5kk8apUQE-120... ). So why does it perform so poorly? Because everything AI-related has been developed and optimized to run on Nvidia's CUDA.

Mostly, this is a mindshare issue. If Intel offered a workstation GPU (i.e. not a ridiculously expensive "enterprise" monster) that developers could use that had something like 32GB or 64GB of VRAM it would sell! They'd sell zillions of them! In fact, I'd wager that they'd be so popular it'd be hard for consumers to even get their hands on one because it would sell out everywhere.

It doesn't even need to be the fastest card. It just needs to offer more VRAM than the competition. Right now, if you want to do things like training or video generation the lack of VRAM is a bigger bottleneck than the speed of the GPU. How does Intel not see this‽ They have the power to step up and take over a huge section of the market but instead they're just copying (poorly) what everyone else is doing.

Going above 24GB is probably not going to be cheap until gddr7 is out, and even that will only push it to 36gb. The fancier stacked gddr6 stuff is probably pretty expensive and you can’t just add more dies because of signal integrity issues.

What is obvious to us, is an industry standard to Product Managers. When is the last time you have seen an industry player upset the status quo? Intel has not changed that much.

"It would be poetic to see 32-48GB at a non-eye-watering price point."

I heard some Asrock motherboard BIOSes could set the VRAM up to 64GB on Ryzen5.

Doing some investigations with different AMD hardware atm.

AMD making drivers of high quality? I would pay to see that :-)

First crypto then AI, I wish GPUs were left alone for gaming.

The issue from the manufacturer's perspective is that they've got two different customer bases with wildly different willingness to pay, but not substantially different needs from their product. If Nvidia and AMD didn't split the two markets somehow, then there would be no cards available to the PC market, since the AI companies with much deeper pockets would buy up the lot. This is undesirable from the manufacturer's perspective for a couple reasons, but I suspect a big one is worries that the next AI winter would cause their entire business to crater out, whereas the PC market is pretty reliable for the foreseeable future.

Right now, the best discriminator they have is that PC users are willing to put up with much smaller amounts of VRAM.

> Intel definitely seems to be doing all the right things on software support.

Can you elaborate on this? Intel's reputation for software support hasn't been stellar, what's changed?

still wondering why we can't have gpu's with sodimm slots so you can crank the vram

There's the Max GPU (Ponte Vecchio), their datacentre offering, with 128GB of HBM2e memory, 408 MB of L2 cache, and 64 MB of L1 cache. Then there's Gaudi, which has similar numbers but with cores specific for AI workloads (as far as I know from the marketing).

You can pick them up in prebuilds from Dell and Supermicro: https://www.supermicro.com/en/accelerators/intel

Read more about them here: https://www.servethehome.com/intel-shows-gpu-max-1550-perfor...

For consumer stuff there's the Intel Arc A770 with 16GB VRAM. More than that and you start moving into enterprise stuff.

You can already use intel GPUs (both ARC and iGPUS) with llama.cpp on a bunch of backends:

- SYCL [1]

- Vulkan

- OpenCL

I don't own the hardware, but I imagine SYCL is more performant for ARC , because it's the one intel is pushing for their datacenter stuff

[1]: https://www.intel.com/content/www/us/en/developer/articles/t...

Intel GPUs got quite a bit of penetration in the SE Asian market, and Intel is close to releasing a new generation. In addition, Intel's allowing for GPU virtualization without additional license fees (unlike Nvidia and GRID licenses), allowing hosting operators to carve up these cards. I have a feeling we're going to see a lot more Intel offerings available.

No, but for consumers they're a great offering.

16GB RAM and performance around a 4060ti or so, but for 65% of the price

Lots offer Intel CPUs though...