Ask HN: Has anyone deployed LLMs to production?

I was having trouble getting GPT-4o to extract data like address, email, phone, tracking number from random emails in an inbox. Sometimes it would do it perfectly and other times it would fail miserably on a similar email. Then I tried asking it to first markup the email with schema.org metadata. Then I asked it to extract the data from the schema.org markup. That worked nearly every time.

Maybe there is an extra step you can work into your prompt that would help it get to the proper classification

They're great at first level customer service. Lots of questions are repetitive and they go through this better than humans. It was the biggest boost to customer satisfaction rating.

On the other end, I actually canceled a $100/month subscription once through email (it was company email that I no longer had access too). Gave evidence. It canceled the subscription within 20 mins.

Also gemini flash is unreliable. The best cost efficiency today seems to be gpt-4.1. The cheaper models seem to be okay for summarization mostly. Gemini Flash was much better a year ago, still unreliable, but at least it followed instructions.

We are using over 50 billion LLM tokens for NLP/classification purposes per month. A mix of self hosted and cloud hosted models. But I have not attempted any fine tuning. Just prompt, (and perhaps more importantly) context “engineering”.

I have Microsoft's Phi4 deployed onto https://mapleintel.ca for the AI side. Currently over 44,000 ips in that list.

I tried 'reasoning plus' but it was so much slower.

We use it heavily for doc search. We bought Kapa.ai a few years ago and leverage their solution, not an in-house build.

Did u ever try to fine tune with openai? it works good for me