Show HN: Carton – Run any ML model from any programming language (opens in new tab)

For exmaple, if your model contains arbitrary Python code, you'd pack it using [1] and then you could load it from another language using [2]. In this case, Carton transparently spins up an isolated Python interpreter under the hood to run your model (even if the rest of your application is in another language).

You can take it one step further if you're using certain DL frameworks. For example, you can create a TorchScript model in Python [3] and then use it from any programming language Carton supports without requiring python at runtime (i.e. your model runs completely in native code).

[1] https://carton.run/docs/packing/python

[2] https://carton.run/docs/loading

[3] https://carton.run/docs/packing/torchscript

The currently supported platforms [1] were mostly driven by environments I've seen at various tech companies.

I do have active plans to support inference from WASM/WebGPU so maybe that could be a good entrypoint to Windows support.

--

[1] Currently, the supported platforms are:

* `x86_64` Linux and macOS

* `aarch64` Linux (e.g. Linux on AWS Graviton)

* `aarch64` macOS (e.g. M1 and M2 Apple Silicon chips)

* WebAssembly (metadata access only for now, but WebGPU runners are coming soon)

> ONNX converts models while Carton wraps them. Carton uses the underlying framework (e.g. PyTorch) to actually execute a model under the hood. This is important because it makes it easy to use custom ops, TensorRT, etc without changes. For some sophisticated models, "conversion" steps (e.g. to ONNX) can be problematic and require additional validation. By removing these conversion steps, Carton enables faster experimentation, deployment, and iteration.

> With that said, we plan to support ONNX models within Carton. This lets you use ONNX if you choose and it enables some interesting use cases (like running models in-browser with WASM).

More broadly, Carton can compose with other interesting technologies in ways ONNX isn't able to because ONNX is an inference engine while Carton is an abstraction layer.

It basically runs with the promise that you can package CUDA / PyTorch / Python interpreter into the host language in some way, and use it.

This is true for Android, not true for iOS, true for almost all desktop systems, somewhat true for web (packaging PyTorch + Python interpreter in WASM, the latter is easy, the former, I am unsure), probably not true for FAAS environments (such as Cloudflare worker, or AWS Lambda).

This lets us do things like fetch model metadata [1] for a large remote model, by only fetching a few tiny byte ranges instead of the whole model archive.

It also means you can include sample data (images, etc) with your model and they're only fetched when necessary (for example with stable diffusion: https://carton.pub/stabilityai/sdxl)

[1] https://carton.run/docs/metadata

To give two examples of prior art, it worked for Quake 3 data files (.pk3) & geospatial data files (.kmz)

Maybe it's not the best choice but it doesn't seem like a bad one.

It's hardly revolutionary to do this, here are some common examples of things that are zip files but don't label themselves as such:

- .jar

- .odt, .ods, .odp, .docx, .xlsx, .pptx

- .epub

- .apk

- .crx, .xpi

We all think this. My initial thought was that this is probably a startup selling PyTorch-as-a-Service, and I did not bother to read the article. It turns out that I was wrong, and this might even be useful -- if not for the implementation, then perhaps for the idea.

However, it turns out to make Hacker News a nicer space if we follow these guidelines:

> Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.