One roadblock I haven't been able to overcome is integrating Nix in CI. In my case, I use CircleCI. I know there are Nix-centric CI platforms, but I've invested a lot into CircleCI and would like to stay.
The problem I hit is that if I switch from something like a sqlfluff Docker container to a Nix Docker container, my build job goes from 15s to 2m+ because now I have to download several gigs within the CI job and build my Nix environment from scratch.[1]
I tried Cachix, but the best I could get was bringing the job down to 55s (a roughly 3x performance penalty from an off-the-shelf Docker container).[2]
How does FlakeHub Cache's performance in CI compare to the default Nix cache or to Cachix?
[0] https://mtlynch.io/notes/nix-dev-environment/
[1] https://github.com/cachix/cachix/issues/579
[2] https://github.com/cachix/cachix/issues/579#issuecomment-175...
With "it works so well", I mean:
* CI is cheap because namespace.so runners are faster & cheaper than GitHub Actions
* the UX is exactly the same as using vanilla GitHub Actions, namespace.so's runner is a drop-in replacement
* the CI starts up very fast because `/nix/store` is stored on namespace.so's "Cached Volumes" which are like virtual disks attached to the CI runner, so there is nothing to download to prime the cache, Nix can just start running immediately
* https://github.com/cachix/cachix-action uploads whatever Nix builds to Cachix in the background so from this point on if any of our devs pulls the latest code, they don't have to build any derivations locally, and lose time waiting, everything is pulled automatically from Cachix to their machines, they can get right to work. Or if I have to purge the Cache Volume for some reason, nothing needs to be rebuilt, just pulled from Cachix.
It's taken a few years to get this dialed in, but now that it works, it's sooo good!
(full disclosure: Determinate Systems employee)
You indeed need some kind of nix-aware runner (e.g. with a persistent /nix) if you want both the flexibility and performance...
I think the fact that there is no bidding process for agents is a mistake that we need to correct. It's a little used concept in distributed computing, but a powerful one when applicable, and I think the preconditions for a fast build, with all of the docker images and other bootstrapping pipelines is making that cost harder to ignore. We are de facto using heterogeneous networks of worker processes now, and it's starting to reach 8 Fallacies level.
You build a distributed queue for response time or throughput, and we are using the throughput model, more or less, while CI/CD is mostly about response time.
I use Gitlab CI and with self hosted runners I can make sure the /nix/store is persistent across CI jobs and then my builds take ~5 seconds if no code has changed, 5 seconds of which is just nix build evaluating all my nix files and realising no work needs to be done.
Is there an easier way to do it?
Also, wouldn't it be pretty similar performance-wise to caching the /nix path on a generic Nix image? I've tried that, but it has problems because the Nix path ends up being a few GB, so that ends up adding a long time to the build as well.
I have the complication of android + Rust (com[compiling rust for android + SQLite require gimnastics: https://www.reddit.com/r/rust/comments/1872ftd/exist_a_solid...).
My ideal setup is to have my dev env be the CI env, to stop the chance that every so often things break (my CI breaks my delicate android incantions with some regularity).
I'm having trouble seeing the audience for this product. If I'm in an org going for SOC 2, it's likely I already have a story around artifact access that isn't intrinsically tied with my build system. If I weren't already building sensitive things with Nix, this doesn't seem like the thing that would get me to switch. If I have a monorepo, flakes are likely off the table right away due to the performance hit. Am I missing something about how people are using flakes right now?
Yes. Lots of orgs are using Nix today and need to meet SOC2 or better, and are using many different caches to meet compliance objectives. This is creating a significant performance penalty.
Further, whether you care about the separation or not the onboarding process is simply nicer, because it is tied to the environment you're running in (GHA) not tokens you have to copy paste around.
> If I have a monorepo, flakes are likely off the table right away due to the performance hit.
If you've heard of "lazy trees", that work is being spearheaded by us at DetSys to address this problem :).
I think I'm still missing something. By all accounts, I and my organization should be in your target market, but there are a few barriers:
- I've achieved significant adoption of Nix for a great many things, but not for building proprietary software.
- I already have an enterprise-grade artifact store to use as a cache.
- I also have an identity platform and robust secrets management.
- I have evaluated lazy trees Nix at a few different intervals and did not find it enough of a speedup to continue using it.
The sum total of all of these barriers is that I may be wrong about whether we're truly in your target market, but I also think those will be common barriers for you, which makes me think I still don't quite understand the product or the market correctly.
Regarding .dotfiles
It seems that there is a push or desire to have everything declared via nix and/or flakes. What I rarely see incorporated is 'git clone' or 'stow' for dotfile management. My simple question is: why not both? What is so wrong with using a 3rd party system as part of the deployment.
Let's say I want to fiddle with my vim/neovim .rc file. If I change it 10 times in the course of an hour, and run 'nixos rebuild switch' (did I get that right?) each time to set it, then I'm going to be spending a bunch of time/CPU cycles rebuilding world.
It seems to me a hybrid approach would work best. -nix rebuild scratch for OS-level modifications -git push/pull/clone for frequently modified files.
Nobody would ever suggest using the nix approach for storing their 'recipe folder'.
Some people do use nix just for their system config and manage their dotfiles separately, but for me it's simpler to just have one system that installs the software I want and configures it the way I like it, rather than two things I'd need to keep in sync with each other.
