undefined | Better HN

0 pointsmlazos2y ago0 comments

Peak H100 power consumption is 700W. Average power consumption of the groq card (from their own website) is 240W. With 576 chips it just doesn’t look good. How much is that millisecond perf gain worth it to end users?

That said I think their arch is super interesting. I just think that demo was way too hype when the actual system is pretty impractical.

0 comments

2 comments · 1 top-level

ein0p2y ago· 1 in thread

So? They aren't performing the same computation. You can't compare the two. What you can compare is power draw at an equivalent tokens/sec on the same model for the entire system. But you don't have that number.

mlazosOP2y ago

I’m just estimating here from public numbers. My point is the power consumption could be 100W on that workload and the groq chip could be 1k, both ridiculously optimistic. The whole system is still crazy expensive. H100s will not have latency as fast, but in terms of concurrent users and TCO, I really don’t think groq will be worth it. You could probably get the same concurrent users and throughput with like 8 h100s. Latency won’t be as good, but price could be much lower.

j / k navigate · click thread line to collapse