undefined | Better HN

0 pointsanonym292y ago0 comments

When you can sell an enterprise-grade card with 40-80GB of VRAM for $50k, selling consumer cards with 24GB for $2k is almost a form of charity, by comparison.

AMD and Intel GPUs do not have the software ecosystem for AI workloads that Nvidia does, though AMD is rapidly improving. Nvidia has had an effective monopoly on the AI hardware space for the last year or so, and continues to have an effective near-monopoly, but that won't last forever as AMD and Intel catch up.

The VRAM is one of the largest differentiators of their cards. Sufficient VRAM allows you to run huge LLMs like 65B in-memory, which is orders of magnitudes faster than system RAM + CPU. Smaller amounts of VRAM require swapping between VRAM and system RAM and incur a major performance penalty.

Businesses are fighting to fork over $50k+/card for 40/80GB cards with the same processor as the 24GB consumer cards - it doesn't make economic sense for Nvidia to offer more on the consumer cards, lest they start cannibalizing demand for the enterprise cards.

0 comments

3 comments · 1 top-level

dragonwriter2y ago· 2 in thread

> When you can sell an enterprise-grade card with 40-80GB of VRAM for $50k

ADA 6000 RTX (48GB) — an enterprise (workstation) cars — is about $10K sticker price. The differentiator between those and data center cards in the 40GB-80GB range is, obviously, not just VRAM.

anonym29OP2y ago

Check out what H100's are going for.

Also, as samspenc points out, that RTX 6000 is using the same AD102 chip found in a consumer RTX 4090, just with marginally more CUDA cores, TMUs, ROPs, etc.

The most substantial difference between the two is that the $10k card has twice the VRAM of the $2k card.

I know it sounds outrageously oversimplified, but Nvidia can indeed more or less print money by attaching a few extra memory chips to what is otherwise a flagship consumer-oriented graphics processor.

The profit margins on the H100, for reference, are estimated to be around one thousand percent (1000%), i.e., they sell for ~10x as much as it costs Nvidia to make them, and demand for them still grossly exceeds the total supply. See: https://the-decoder.com/nvidias-h100-gpu-sells-like-hot-cake...

samspenc2y ago

And yet if you look at Blender (open source & free) GPU benchmarks https://opendata.blender.org/benchmarks/query/?compute_type=...

The RTX ADA 6000 actually slightly performs worse than a consumer-grade 4090 (~10% less performant), even though it retails for 4-5x more ($8000 for ADA 6000 compared to $1500-2000 for a 4090).

j / k navigate · click thread line to collapse