egr

I am currently designing a university course on ML-Product Management.

The module's focus is to teach my students that when it comes to building efficient and effective solutions the problem-analysis comes first and the ML-side is just a part of the overall challenge. To that end the student teams collaborate with industry and government institutions to develop solution designs and prototypes.

Some of the best practices we apply are based on my experience as a CSO in the contract intelligence space:

- solution scoping based on task complexity, open-ness of world model, and dynamics of environment - business metrics first, ideally correlate ML-metrics to business metrics - fast iteration and evaluation on the model level - fast wire framing and prototyping to expose the ML-based functionality to end-users

I am curious how other's approach this task? What is your approach, what literature do you recommend?

How do you guys approach decision making when it comes to ML-powered solutions?

The approach I have followed so far in my career follows decision-making based on collecting evidence as quickly as possible based on evaluating the performance on the ML-level (anecdotal to iterative creation of test sets), rapid prototyping on the user interface and UX level (as most real-world production systems require a human-machine interaction approach), and building functional PoCs as soon as possible.

Based on as much insight as can be collected in a period of weeks to sometimes several months we then try to make a GO/No-GO decision if the solution is feasible and has a good return on investment.

I am extremely curious how others approach this decision-making task.

I was wondering what is HN's take on the state of the art when it comes to hosting GPU-based Machine Learning model demos. We are a university unit with a strong focus on Applied ML and the need to host research applications and student project applications that rely in GPU-based models.

We currently focus on the automation of the CI/CD of these demo applications and prototypes that are based on computer vision models or LLMS. We are aware of solutions that cover the pipeline aspect of CI/CD such as Gitlab, Github and the ML OPS platforms.

Where it currently breaks down for us, is the identification of hosting that would allow us to deploy a range of demo or prototype applications.

Our requirements are:

- host a number of applications / demos at the same time

- only prediction is of interest here (training happens on other machines)

- in many cases there is the need to host our own trained models

- GPU in the range of 12-24 GB GPU

- traffic would be minimal

- minimum redundancy or fault-tolerance levels

- would like to keep expense at level of 500-1000$ per month

The hosting platforms we are currently evaluating are:

- https://www.centron.de/en/centron-cloud-gpu-services/ - https://www.ovhcloud.com/en/public-cloud/prices/

To use these platforms to host multiple demos would it be the easiest to run Kubernetes and deploy Pods? Are there good best practices to host multiple low traffic web applications based on this setup? Or maybe there exists a much simpler setup to share a GPU between applications.

Thanks a lot in advance for your input. This will directly flow into the hosting implementation but will also find its way into the teaching after review.