No comments yet.
TL;DR: We’ve launched StableBuild, a new tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
As the CTO at an ML startup w/ 75 people (https://edgeimpulse.com/) I’ve grown incredibly frustrated with non-deterministic builds. Last year basically every week one of our containers (we have 40+ unique ones in prod) would stop working properly because some dependency was updated or removed. This ranges from Nvidia deleting cuda base images from Docker Hub, to Chromium being removed from the Ubuntu package registry in favor of the snap version, to pandas 2 being published with breaking APIs - while everyone just depends on e.g. pandas>=1.4.
This has been super disruptive because builds break for no apparent reason: someone pushes some unrelated code change, a container needs to be rebuilt, now it gets the latest dependencies => boom, either a compile error or an integration test fails. Many times this even blocks deployment. If the build system has decided that a container on master needs to be rebuilt, we can’t deploy the complete system if a dependency has shifted. And, fixing this naturally falls on the most senior engineers.
Anyway, to fix this I’ve funded (together w/ my Edge Impulse cofounder) StableBuild. It’s a set of mirrors and registries that let you easily freeze and pin Docker images, apt/apk packages, Python packages, and arbitrary files and URLs from the internet. It currently consists of:
* A custom pull-through cache for Docker Hub, that makes any image pulled immutable. Protects against updated or removed images; and as a nice byproduct also bypasses pull-rate limits in Docker Hub.
* Full daily copies of the Ubuntu, Debian and Alpine package registries + the most popular PPAs; so you can pin to a specific date (give me the package registry as it was on 2023-12-15). Essentially what snapshot.debian.org does, but fast and highly available (and for more repos).
* Full daily copy of the PyPI registry, so you can also pin to a specific date. This has been super useful for resurrecting old Python code. Any Python example w/ dependencies is bitrotted the moment it gets published - StableBuild’s historic registry helps tremendously (see https://docs.stablebuild.com/mirrors-and-caches/pypi-mirror#...)
* A generic file / URL cache for arbitrary things you need to pull from the internet during builds.
This has all been in production with SB’s first customers and has basically eliminated random build failures due to changed dependencies for them. Naturally you still want to upgrade dependencies (security patches are nice!) - but you can do it at their own pace, rather than whenever a container rebuilds.
Long story short: we’re releasing StableBuild to the public today. Pricing starts at $199 / month - which yes, is definitely more than free, but is cheaper than just running your own historic apt mirror on Amazon (which we used to do at Edge Impulse) - and there’s a 7 day free trial (CC needed). Would love to hear people’s thoughts :-)
Sign up: https://dashboard.stablebuild.com
