A PyTorch Approach to ML Infrastructure (opens in new tab)

Hi! That's awesome to hear, and very aligned with the devx we're going for. How was your system received?

In fact we totally agree and are not cloudpickling the function because of the package minor version issues. We sync over the code to the destination environment and the server imports it fresh, which is much more robust. The one piece of code that cloudpickles functions is a trap door for certain weird situations, but frankly we haven't had to use it in months.

It's a great point. The funny thing is that the rest of the world just has dev, QA, and prod staging and canaries, while ML has "the 6 months it takes to translate from notebook to pipeline" or "uploading a new checkpoint/image to the platform." We can just stage properly and release through CD like everyone else does, but not spend 6 months flipping the switch. We built the ability to specify the package for a function as a git repo and a particular revision to enable this, and hopefully it means more people rely on version control as the source of truth in prod and not the most recently uploaded model checkpoint to Sagemaker. During experimentation though, it really is frustrating that many systems only allow you run what you've committed.

We've also built a basic permissioning system to control who can actually overwrite the saved version of a resource, so there are no accidents. E.g. if the prod inference blob is saved at "mikes_pizza/nlp/bert/bert_prod", you can set it so only x accounts can overwrite that metadata to point to a new model. Ideally we just inherit existing RBAC groups sometime soon.

Does that make sense? Curious if you had something else in mind as far as the danger.

Yes, we work pretty closely with them and they're lovely. Everyone should try SkyPilot.

Good catch - this is actually only for functions defined inside interactive Notebook or iPython environments, and we do have an option to bypass it (which serializes the function and state), but you probably don't want it by default. Any function defined inside normal python (even if you're using it within a notebook) doesn't doesn't need imports or variables in scope, you can ship it as is to remote compute. But notebooks are gnarly things, where you might have defined a variable and changed it many times, and finally used it inside a function which you want to run remotely as a service. You probably don't want that variable's state to be shipped over, and PyTorch doesn't do this either. This is why so much of stateful PyTorch is meant to be defined inside new nn.Module classes, to neatly package the state to send over to GPU memory. We offer more flexibility than that, but the more state you ship over, the more likely you are to hit version conflicts between the local and remote environments, which can be really annoying. We practically never run into those issues at Runhouse nowadays because we think we've found a sweet spot.

Good question. We actually use Ray to handle a bunch of the scheduling within the compute, but largely see our role as outside the compute. Meaning, Ray provides a powerful DSL for distributed compute, while we are aggressively DSL-free so users can ship Ray code, PyTorch Distributed, Accelerate, Horovod, etc. to their hardware through Runhouse. We're more focused on connecting disparate compute and storage and making them multiplayer (but largely see the cluster as an opaque unit of compute) while they're more focused on enabling distribution inside the cluster, if that makes sense.

No, I know of CentML but don't deeply know the surface of hardware they compile for. I'm enthusiastic about projects like this and others which integrate with PyTorch 2.0. Flexible compilers make the value of being able to ship your code around to various hardware even more powerful.

That's a good question. I actually love the Mojo concept, but see it as very different. They're creating a portable acceleration option in Python proper, while we're trying to make it so you can easily ship around such code to different infra. You can see them or other DSLs like Ray as handling "inside the cluster" while we're focused on solving "outside the cluster." That's what I've picked up from their marketing but could be missing something.

In general making code itself more portable is great (which is the objective of many ML compilers) and will make Runhouse even more valuable, because the ability to take the same code and send it to different places shines when those different places can be different compute types.

Looks like Runhouse is FOSS (Apache 2.0) and you get to choose your own infrastructure. I will try out Runhouse. Mojo wants me to send them my info to get started.

What is the mojo approach?