Ask HN: How do you track NLP models in prod?

1 pointsNevin19012y ago5 comments

Working on a service which uses GPT-3.5 and was wondering how I would go about tracking the usefulness of it. Wondering what you guys have done, or if you guys even track your NLP models at all.

5 comments

3 comments · 1 top-level

PaulHoule2y ago· 2 in thread

Are you doing classification, regression, extraction or something different? How did you evaluate the model before it went into production?

Nevin1901OP2y ago

Causal text generation. I evaluated it on my own internal dataset but real world data shifts over time.

PaulHoule2y ago

How do you get a number for text generation?

1 more reply

j / k navigate · click thread line to collapse