undefined | Better HN

0 pointslatchkey22d ago0 comments

Not sure I understand. If they support MI300x, their self-hosted will run on our hardware.

0 comments

2 comments · 1 top-level

erichocean15d ago· 1 in thread

If it was that easy, I wouldn't have commented.

It's not, which is why it would be nice if they did the actual work (on your hardware).

I would 100% pay $16/hr to run a self-hosted instance, but I won't spend thousands of dollars to (maybe) get it working (my time + the hardware).

latchkeyOP15d ago

Ok, sure. Valid. Have you asked them to support V4?

https://docs.modular.com/max/models/

I agree with you though, serving up inference is secret sauce for a lot of teams and not everyone publishes how to do it because of the costs involved in doing so. They need an ROI.

j / k navigate · click thread line to collapse