Ask HN: Anyone have dedicated, local, AI server?

3 pointsskwee3571mo ago1 comments

I'm toying with the idea of trying to run some local AI models for both agentic coding, as well as trying to get into developing products on top of AI models.

I'm not, yet, looking into training models, but more about inference, while staying relatively on a budget. I have about €2-3k to spend on hardware, for a server I can run at home. I know that one can rent an AI inference server, but from my research, they tend to cost around €200-300/mo, which means that my hardware will pay for itself in ~10 months.

However, I'm quiet confused with what hardware to chose. I saw recommendations for Ryzen AI Max+ 395, in SFF PCs, but some people suggest instead to build a "regular" PC with Nvidia 3090 or 4090.

People who run small servers (as opposed to €10-15k beasts), what's your experience and recommendations?

1 comments

lesserknowndan1mo ago

Do you have a spare computer that you can load up with RAM so that you can try out some available models? It may not be fast, but it will give you an idea of how capable the models are.

I have been running some local models on LMStudio using an AMD 5950XT with 128 GB Ram (until recently with no GPU offloading). My two cents is that I keep reaching for my free ChatCPT account because I find the local models fairly unreliable.

Some just seem to get into a thinking loop where they go over the same ground again and again. Others can just output pure garbage.

Would anyone here suggest any specific models. Especially for Swift/SwiftUI. PHP. And MySQL Stored Procedures.

j / k navigate · click thread line to collapse