undefined | Better HN

0 pointsfrognumber1y ago0 comments

Assuming you want to maintain full bandwidth.

Which I don't care too much about.

However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards.

If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it will be the cheapest way to run models.

0 comments

0cf8612b2e1e1y ago

At this point, I cannot run an entire class of models without OOM. I will take a performance hit if it lets me run it at all.

I want a consumer card that can do some number of tokens per second. I do not need a monster that can serve as the basis for a startup.

hnfong1y ago

A maxed out Mac Studio probably fits your requirements as stated.

0cf8612b2e1e1y ago

If I were willing to drop $4k on that setup, I might as well get the real NVidia offering.

The hobbyist market needs something priced well under $1k to make it accessible.

rnewme1y ago

How comes you don't care about full bandwidth?

frognumberOP1y ago

Mostly because I use this for development.

If a model takes twice as long to run.... I'll live. Worst-case, it will be mildly annoying.

If I can't run a model, that's a critical failure.

There's a huge step up CPU->GPU which I need, but 3060 versus 4090 isn't a big deal at all. Indeed, the 24GB versus 16GB is a bigger difference than the number of CUDA cores.

Dalewyn1y ago

The thing about RAM speed (aka bandwidth) is that it becomes irrelevant if you run out and have to page out to slower tiers of storage.

j / k navigate · click thread line to collapse

0 pointsfrognumber1y ago0 comments

Assuming you want to maintain full bandwidth.

Which I don't care too much about.

However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards.

If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it will be the cheapest way to run models.

0 comments

0cf8612b2e1e1y ago

At this point, I cannot run an entire class of models without OOM. I will take a performance hit if it lets me run it at all.

I want a consumer card that can do some number of tokens per second. I do not need a monster that can serve as the basis for a startup.

hnfong1y ago

A maxed out Mac Studio probably fits your requirements as stated.

0cf8612b2e1e1y ago

If I were willing to drop $4k on that setup, I might as well get the real NVidia offering.

The hobbyist market needs something priced well under $1k to make it accessible.

rnewme1y ago

How comes you don't care about full bandwidth?

frognumberOP1y ago

Mostly because I use this for development.

If a model takes twice as long to run.... I'll live. Worst-case, it will be mildly annoying.

If I can't run a model, that's a critical failure.

There's a huge step up CPU->GPU which I need, but 3060 versus 4090 isn't a big deal at all. Indeed, the 24GB versus 16GB is a bigger difference than the number of CUDA cores.

Dalewyn1y ago

The thing about RAM speed (aka bandwidth) is that it becomes irrelevant if you run out and have to page out to slower tiers of storage.

j / k navigate · click thread line to collapse