undefined | Better HN

0 pointsjrockway1y ago0 comments

I think this, or something of equal complexity, is probably the right choice. I have spent a lot of time helping people with their dev environments, and the same problems keep coming up; "no, you need this version of kubectl", "no, you need this version of jq", "no, the Makefile expects THIS version of The Silver Searcher". A mass of shell scripts and random utilities was a consistent drag on the entire team and everyone that interacted with the team.

I ended up going with Bazel, not because of this particular problem alone (though it was part of it; people we hired spent WEEKS trying to get a happy edit/test/debug cycle going), but because proper dependency-based test caching was sorely needed. Using Bazel and Buildbuddy brought CI down from about 17 minutes per run to 3-4 minutes for a typical change, which meant that even if people didn't want to get a local setup going, they could at least be slightly productive. I also made sure that every dependency / tool useful for developing the product was versioned in the repository, so if something needs `psql` you can `bazel run //tools/postgres/psql` and have it just work. (Hate that Postgres can't be statically linked, though.)

It was a lot of work for me, and people do gripe about some things ("I liked `go test ./...`, I can't adjust to `bazel test ...`"), but all in all, it does work well. I would do it again. Day 1 at the company; git clone our thing, install bazelisk, and your environment setup is done. All the tests pass. You can run the app locally with a simple `bazel run`. I'm pretty happy with the outcome.

Nix is something I looked into for our container images, but they just end up being too big. I never figured out why; I think a lot of things are dynamically linked and they include their own /usr/lib tree with the entire transitive dependency chain for that particular app, even if other things you have installed have some overlap with that dependency chain. I prefer the approach of statically linking everything and only including what you need. I compromised by basing things on Debian and rules_distroless, which at least lets you build a container image with the exact same sha256 on two different machines. (We previously just did "FROM scratch; COPY <statically linked binary> /app; ENTRYPOINT /app", but then started needing things like pg_dump in our image. If you can just have a single statically-linked binary be your entire app, great. Sometimes you can't, and then you need some sort of reasonable solution. Also everything ends up growing a dependency on ca-certificates...)

0 comments

pxc1y ago

> I think a lot of things are dynamically linked and they include their own /usr/lib tree with the entire transitive dependency chain for that particular app, even if other things you have installed have some overlap with that dependency chain. I prefer the approach of statically linking everything and only including what you need.

Wherever there's such overlap, those dependencies are already shared. Static linking in such a situation means more disk usage, not less.

Packages in Nixpkgs have large closure sizes for entirely other reasons, like not splitting packages as aggressively as they could be split, or enabling/including most optional dependencies by default. Distros like Alpine typically lean the other way for their defaults.

It's true that if you're willing to manually mangle them, static binaries are nice because you can very easily strip all docs and examples or even executables that you don't need, and still know your executable will have the libs it's linked against. In one place at work I actually do this with Nixpkgs— there's a pkgsStatic that includes only statically compiled packages. I pull just the tiny parts of some package I need out and copy them onto a blank OCI image because it was the path of least resistance.

But Nix also has some really nice tools for inspecting dependency graphs to figure out why large packages are getting pulled in. nix-tree is my favorite, but there's also the older nix-du that gives the same info via graphviz instead of the terminal, and the built-in `nix why-depends`.

-----

Edited to add: wait, are you saying you used some other base distro to create Docker images where some things were supplied by Nix and others came from the base distro? If so, yeah, Nix is going to bring all the dependencies along, all the way down to libc or whatever. That's required for the kind of hermeticity that is its goal.

Mixed images like that are always going to be larger. But you also don't need a base distro at all with Nix. You can use one of the existing Nix libraries for Docker/OCI stuff to generate a complete image from scratch, or just copy your Nix packages' dependency closure onto an empty image with a FROM SCRATCH Dockerfile.

If you can't do that, you can do various things to try to slim things down but it's best to just Nixify whatever other packages you're using so you don't need a base distro. (And if you're trying to save space, Nix itself doesn't need to be in your Docker images either, which can also cut out some deps.)

jrockwayOP1y ago

Just to answer your edit, nope, I wasn't mixing distros or anything. It would either be all Nix or all Debian.

pxc1y ago

Got it. Yeah, that would likely be easiest to resolve by exploring the closure with nix-tree and looking at how to eliminate or split dependencies. But pkgsStatic can be a handy fallback option too.

j / k navigate · click thread line to collapse

0 pointsjrockway1y ago0 comments

0 comments

pxc1y ago

Wherever there's such overlap, those dependencies are already shared. Static linking in such a situation means more disk usage, not less.

-----

jrockwayOP1y ago

Just to answer your edit, nope, I wasn't mixing distros or anything. It would either be all Nix or all Debian.

pxc1y ago

Got it. Yeah, that would likely be easiest to resolve by exploring the closure with nix-tree and looking at how to eliminate or split dependencies. But pkgsStatic can be a handy fallback option too.

j / k navigate · click thread line to collapse