We had a team of data scientists who were developing models (in the abstract sense) that we would then train on specific and then deploy as concrete models.

This was just before the BERT paper came out, we were working with a few different models, sometimes the classical models from scikit-learn, sometimes these were

https://en.wikipedia.org/wiki/Convolutional_neural_network

for text trained with a GPU. Ideally I wanted to make it easy for the data sci's to put their models in front of customers so it was not about getting one model up and running but developing a process to grease the skids. Back then, for instance, a particular version of Tensorflow required that you had a particular version of the CUDA libraries installed. We'd start with models we downloaded off the net so it was important that we could support whatever version of Tensorflow the model needed and have to click-click-click on various permission forms the way NVIDIA insisted on. (Turned out the CUDA libs are just userspace libraries and you can pack them up in a wheel)

There was that and the fact that pip's resolving algorithm was and is incorrect (e.g. works OK all the time for very simple project, screws up occasionally for medium sized projects but it is easy to work around and maker excuses for, a big project might fail to build with complete reliability) that you can sabotage all your virtualenvs with this one weird trick

  pip install --user <some_package>

that data sci's often think they are too smart to have to have any discipline, that Docker is part of the problem and not part of the solution when it means the data sci's now can easily find a Python where the default charset is Hungarian, etc.

After seven months or so I'd figured out close to 100% of the reason why our Python builds (really anybody's Python builds) were not reliable but the wetware problems in our organization and outside of it were severe. (If there was one root cause for "python builds being unreliable" it is that the Python community was willing to live with unreliable pip for so long)

Management though was sick and tired of Python and we were also focused on other aspects of our value prop so we quit working on the Python trainer and they had me working on another part of the system in Scala + Typescript. (There it was frustrating that we had a data analysis pipeline that didn't give the right answers consistently because even though we'd thought a lot about how to initialize it nobody had thought about the problem of how to tear it down)

I thought it was quite "incredible" (a favorite word of the CEO) when it was announced they'd been bought by one of the world's biggest footwear brands but it was true -- I really did believe in our vision and told everyone that I thought our product could make so much value for one of our customers that they'd buy us and that happened.