The area that needs improvement is the documentation. Once you learn the Nix language, reading the source code is pretty helpful, but it would be nice to make it more approachable. For example, the nixpkgs repo has a bunch of Nix helper functions that are useful to developers when writing their own packages, but these functions' documentation is either buried in a long manual, or non-existent.
Rez ( https://github.com/nerdvegas/rez ) is used by some visual effects companies (including mine) to manage software packages.
It allows us to have great flexibility in the mix & match of software versions and runtime environments.
For Nix, this is almost a side-effect, however. Nix's primary task is to describe how to build packages, and the entire OS is just another package that groups all dependencies. You can make a single change to a file everything depends on and it will result in a new OS.
Nix can evaluate the entire system configuration in seconds, and build or download missing binaries in parallel.
As a result you can e.g. have a complete OS based on a different glibc (maybe with some patch you like) installed and running alongside the normal OS without the glibc patch. Packages that do not use glibc are simply shared.
So Nix is ultimately a tool for making and sharing reproducible package builds. It has a binary cache, but it's not necessary. Like Ports, packages get built by default.
Docker, on the other hand, is a distribution and execution mechanism. It provides an abstract way to move around a fully assembled, ready-to-go service or application running in isolation.
It's entirely reasonable to use both. You can use Nix to build and manage docker images and make extremely minimalist docker images. You can use Nix knowing that the entire process is perfectly reproducible, and the Docker containerization is only a final integration step.
With this, you sorta get the best of both worlds. You get a reproducible build (and if done right, also a reproducible dev environment via nix-shell) and with Docker you get the ability to build and run a prepped copy with a well-defined interface.
Docker really doesn't provide a way to reproduce a built image from scratch. You sort of have to trust and build on existing images, and most folks making bulk images appeal to external tooling outside docker files to do this.
NixOS committer here.
Docker attempts to achieve reproducibility by capturing the entire state of a system in an image file. It also attempts to conserve space by taking a layered approach to images, so when you base your Dockerfile on some base image, your resulting image is the union of the base image's layers and your own changes.
Here's where Docker's approach falls down, and how this could be fixed (and indeed is, by Nix).
Flaw #1:
Building an image from a given Dockerfile is not guaranteed to be reproducible. You can, for example, access a resource over the network in one of your build steps; if the contents of that resource changes between two `docker build`s (e.g. a new version of whatever you're downloading is released, or an attacker substitutes the resource), you'll silently get different resulting images, and very likely will run into "well, it works on my machine" issues.
Solution:
Prohibit any step of your build process from accessing the network, unless you've supplied the expected hash (say, sha256) of any resulting artifacts.
For the projects I work on in my free time, using NixOS on my personal computers, I've never been bitten by nondetermism.
I wish I could say the same about my work projects that use Docker. My team members and I have run into countless issues where our Dockerfiles stop working and then we have to drop everything and play detective so e.g. new hires can get to work, or put fires out in our C.I. env when the cached layers are flushed, etc. So many wasted hours.
Flaw #2:
What happens if you have two Dockerfiles that don't share the same lineage, but you install some of the same packages? You end up with multiples layers on your disk that contain the same contents. That's wasted space.
Solution:
I'll use NixOS as an example, again. In NixOS, you can look at any package and compute the entire dependency graph. It should be noted that this graph includes not only the names of the packages, but also precisely which version of each package was used as a build input. This goes for both build inputs and runtime dependencies.
Note: by "version" I mean not only the version as listed in the release notes, but every detail of how the package was built: which version of Python was used? And then transitively: what version of C was used to compile that Python? etc, etc.
NixOS exploits this by allowing you to share packages with the host machine, and then each NixOS container you spin up has the necessary runtime dependencies bind-mounted into the container's root file system. As a result, NixOS has better deduplication (read: zero). Also, by the same graph traversal mechanism, it's trivial to take any environment, serialize the runtime dependency graph, and send that graph to another machine, back it up somewhere, create a bootable ISO for CD/USB -- whatever you can dream up.
Thanks to Nix's enforced determinism, you can trivially build container environments (and all of their required packages) in parallel across a fleet of build machines. In fact, Nix's superiority at building packages is so strong that people have gone so far as to build Docker images using Nix instead of `docker` (where they can't avoid Docker entirely for whatever reason): https://github.com/NixOS/nixpkgs/blob/2a036ca1a5eafdaed11be1...
You can read more about the Docker image building support here (along with my own rationale for this in the comments): http://lethalman.blogspot.com/2016/04/cheap-docker-images-wi...
I'm keeping things simple here and trying to address the most salient "Docker vs Nix" points, though I could continue talking about other strengths of NixOS outside of the scope of Docker/container tech, if desired.
Docker's union filesystem approach is great in a world where you can't use a better package manager. For everyone else, there are package managers that obviate the need for such hacks, don't waste space, provide determinism at both runtime and build time, etc.
Nix isn't only a package manager, it is also a functional programming language intended for system administration. This means that, while a Nix file is comparative to a Dockerfile, it has several key differences:
1. All Nix files are just functions that take a number of arguments and return a system config (like a JSON object, but with some nice functionality). A Dockerfile is a set of commands you run to build a system to a "starting" state. This is imperative (you're telling the computer to "do this", then "do that", etc.), so once it's done, you can mutate state to deviate from what you specified in your Dockerfile. With Nix, while you can technically do this on some systems, it does provide you with the command line tools so you don't break things (e.g. nix-shell, nix-env). Note, on NixOS, other measures are taken to encourage safety.
2. If things go wrong in Nix(OS), the idea is that you can do a fresh install in your system, copy your old Nix file to it, and with one bash command, be back to where you were before things went haywire. In terms of containers, there's nothing new with this because this is exactly what Docker does. However, Nix also has this concept of generations, so every time you use a Nix command to change your system state either declaratively via a Nix file or imperatively in the command-line using nix-env, you can roll back to a previous version of state. This is especially nice with NixOS, because it creates generations for your entire system too (includes hardware config, drivers, kernel etc.), and makes separate GRUB entries for each generation, so if something breaks after you do a system upgrade, you just chose an old GRUB entry to go back to where you were. AFAIK, Docker doesn't offer anything like this, and this is a good example of how these tools' designs can impact their feature-set so dramatically.
3. A neat feature of Docker is composability. You can inherit from other, pre-existing Dockerfiles, and you can deploy multi-container apps with various tools. Composability at a single-container level is very straightforward with Nix. Since every config is just a function, you simply call the function exposed by a different Nix file with the correct arguments, and... voila! Once you've made your desired Nix file, you can run it either using nix-shell or nixos-container. While I'm no expert, I believe they perform better than Docker as they don't use virtualization. For multi-container deployment, there is NixOps. You write some Nix files describing the VMs you want to deploy, and run a bash command to deploy them to various back-ends (AWS, Azure, etc.). Again, the big difference here is that you can incrementally modify these VMs in a safe way using Nix. If you change your deployment config file, Nix will figure out something has changed, and modify the corresponding VMs to achieve the desired state.
Some may believe that Docker and Nix are very similar, and to their credit, they are in certain scenarios. The thing I like about Nix is that it's one language (and architecture) that was designed well. It's minimal, yet makes it possible to do so much in a safe way.
Nix has been around for a while, but I think the community is growing quickly as functional programming continues to take off. I'm excited to see where it goes, and am super grateful I have a tool like this to use while coding.
This is a problem with lots of feature-rich software, even with meticulously-documented APIs. What we need is reverse-indexed documentation. That is, an extensive API reference is only useful for someone who already knows what functions are in the API and just needs to remember how to use them. But even the most thorough API reference does nothing to promote discovering new functionality. This is often left to the authors, who then have to go about writing a User's Guide that gradually explains concepts, idioms, etc. in prose.
Thorough User's Guides are rare because they are tough to write, and even tougher to write well. Users don't often have the time to read through potentially hundreds of pages of prose to find what they're looking for. We need a better way to let users search or browse for concepts, and then be given a list of the functions that implement each concept.
That is, addition to documentation like:
size_t strlen(const char * s);
RETURN: Length of string s.
size_t strnlen(const char * s, size_t maxlen);
RETURN: Length of string s, or maxlen (whichever is smaller).
NOTE: Stops reading after maxlen.
char * stpcpy(char * dst, const char * src);
Copy src to dst.
RETURN: pointer to trailing '\0' of dst, or dst[n] if no trailing NUL.
NOTE: Undefined behavior if dst and src overlap.
char * stpncpy(char * dst, const char * src, size_t len);
Copy up to len bytes from src to dst.
RETURN: pointer to trailing '\0' of dst, or dst[n] if no trailing NUL.
NOTE: Undefined behavior if dst and src overlap.
char * strcpy(char * dst, const char * src);
Copy src to dst.
RETURN: dst.
NOTE: Undefined behavior if dst and src overlap.
char * strncpy(char * dst, const char * src, size_t len);
Copy up to len bytes from src to dst
RETURN: dst.
NOTE: Undefined behavior if dst and src overlap.
strcpy
TAGS: "concept":"data type":"text", "concept":"attribute":"length",
".input":"array", ".input":"char", ".input":"pointer",
".return":"string", ".return":"pointer"
strncpy
TAGS: "concept":"data type":"text", "concept":"attribute":"length",
"concept":"data type":"array":"max-length operator",
".input":"array", ".input":"char", ".input":"pointer",
".return":"array", ".return":"pointer"
concept
└─ data type
└─ text
└─ array (conceptual)
└─ max-length operator
└─ attribute
└─ length / size
data structure
└─ char
└─ array (implementation)
└─ pointer
'"concept":"data type":"text" AND "concept":"attribute":"length"'
It seems like a PITA at first, but I'm pretty sure tagging functions in their docstrings is easier than writing a whole new User's Guide.
Looks like the Nix expression language is untyped, so this wouldn't work directly, but maybe adding a rough type signature in the docstring would get some of those benefits (and it should be a bit better for discover-ability, since you wouldn't need to guess the same tags/concept the author choose).
I also think that you could get a lot of people to agree that it is a great idea, and STILL have a lot of trouble enforcing it in a project without very rigid code review policies and very good linting rules.
Isn't all that also true for Guix?
If so, why do your prefer Nix over Guix?
Unfortunately, that isn't workable for most of us.
I cannot answer your question, but I think Nix has (for some reason) a lot more publicity than Guix at the moment.
But the package manager itself might be a good idea for dev environments, but all the languages i use provide similar facility(not counting system-library dependencies).
Many things I got from AUR packages are binary-cached in stable Nixpkgs, not to mention "compilers" (GCC, clang, and various versions of GHC in my case). I don't know when you last tried it out, but binary availability isn't a problem now if you'd like to give NixOS a spin. The only thing I build from source is polybar[0].
I set out with the goal of building a development environment for a software I was working on. I thought - Nix sounds like a better Docker, where it's possible to choose package versions independently of the rest of the system. It's perfect for testing!
I found the package description language refreshing - it was powerful without getting too complicated. Over the course of a couple of days I was able to adapt a few packages and create my own too, with the build environment.
It was almost finished, but the last problem remained: installing CUDA and the userspace portion of the Nvidia driver. It's not rocket science – I thought – all I need is a few .so's, it even works with Docker.
Alas, after battling Nix for another couple of days, trying to use generic GL, installing the Nvidia one, ignoring it altogether or trying to using the host version - I gave up. I found no way to build a package linked against OpenGL that would actually work.
Despite that, I hope to use Nix with a different project in the future.
* https://github.com/NixOS/nixpkgs/issues/9415#issuecomment-30...
https://github.com/pauldotknopf/darch
It is essentially Docker, but you can boot bare-metal with it. It even uses Docker under the hood.
Every boot is a fresh boot. You can read-write during boot, but all changes are made to ram. You mount with fstab things you want persisted, like home.
It uses very simply primitives, which makes it easy to use. Squashfs/grub-mkconfig/overlayfs/etc. Docker isn't exactly a "primitive", but it is easily reasoned with.
I agree that Nix is hard to deal with when it comes to binary distributions of any kind, which includes CUDA.
The learning curve has been absolutely tremendous, however. That said, the support on Freenode has been some of the best I've encountered in more than a decade.
https://nixos.org/nix/manual/#chap-package-management https://rycee.net/posts/2017-07-02-manage-your-home-with-nix...
Do you install software with "nix-env -i package" or do you have a declarative way of doing it? If you use nix-env, how is that better than "brew install package"?
I'm not doubting your setup, I'm just trying to use nix for the same thing and am having a hard time finding the canonical way of doing a declarative user setup.
Declarative or not, I'm still gonna get the advertised benefits of Nix like, say, having some confidence that I got all the dependencies specified properly if my stuff builds, easily customizing packages by overriding specific bits, and coping with arbitrary version requirements really easily.
I'm sure someone who's more used to configuring homebrew has a different perspective there, but it definitely seems preferable to taking over /usr/local and rotating things in and out of there as I switch between projects or w/e.
For me, the biggest one is that I can't use it at clients running a Windows shop. That's a blocker.
I've found Guix and GuixSD, which are a GNU-blessed Guile Scheme-based reimplementation of Nix and NixOS, more aesthetically pleasing. Quite simple and elegant in fact.
The major differences are that Guix DSLs are implemented on top of Scheme, whereas Nix uses a custom DSL. Furthermore, Guix avoids systemd and uses GNU herd instead.
If you just want a solution for your packaging management needs - Nix is definitely not it. Nix is more like a thing to get inspired by to make a package manager.
[1] https://www.reddit.com/r/NixOS/comments/64xyd7/nix_package_m...
It seems a little strange to me to have a functional package manager, but an imperative package install process. I think there are a few projects like NixUP and home-manager that are looking to address this, but I'm hoping Nix will provide an official way to do it.
I basically define a custom meta-package with all the packages I want as dependencies and install it with `nix-env -iA`.
[0]: https://github.com/asymmetric/dotfiles/blob/master/config.ni...
If you click through to https://nixos.org/nixos/nix-pills/why-you-should-give-it-a-t...
you will see "Nix is a purely functional package manager and deployment system for POSIX"
A lot of these systems are just reimplementations tailored to language-X. Nix is pretty unique and clever. For example, you can effectively version control you virtual environment, it's at the OS level, and accommodates major languages and libraries (Python, Perl, Lua, Java, Go, Qt).
Before I get started though, what things could I consider implicit prerequisites of running Nix? For example: I've never built a Debian or RPM package. Would I be better served gaining some background in OS package management before diving into this?
Read the Nix Pills series, that'll teach you more than any amount of playing with non-Nix software.
I can definitely see giving Nix priority for work-related development, but I would like to know if Homebrew can still fill in as a backup option.
Simula is a bleeding edge VR Desktop project for Linux; virtually all of its dependencies are highly novel and require building from source for almost any distro. Nix allowed us to reduce the effort to building our project from 1hr of sifting through build documentation to a single build command.
Nix's only issue is that it so far can't handle OpenGL dependencies very well. I'm not sure if this is something in principle it cannot handle, or if it just hasn't been done yet. I'm praying it's the latter.
Hardware is the one thing that actually varies between systems no matter how declarative your configuration is - this is normally not a problem, since the kernel abstracts it away.
The problem with OpenGL, then, is that every graphics driver provides its own set of OpenGL libraries that is subtly different from all the others. This means that applications need to be built against a specific set of OpenGL libraries from a specific graphics driver, and the kernel is of no help here.
That's fundamentally where the OpenGL issues come from; Nix packages typically expect the libraries to be semi-statefully provided in a `/run` directory. This gets especially hairy when targeting different distros.
EDIT: And because the package doesn't know what graphics drivers it'll be running against on non-NixOS, it can't automatically build against a copy of the correct drivers in nixpkgs either.
There are also bugs. For example, non-NixOS users of Simula are forced to run this script: https://github.com/SimulaVR/Simula/blob/master/swrast.sh
This is because a nix package expects to find a driver in `/run/opengl-driver`, which isn't always present if you're not running NixOS.
You can put that /nix folder anywhere, but you can no longer use their binary package caches and it will recompile everything. This is because they hard-code paths for dependencies during build time.
But, it depends on user namespaces being enabled on your system. That may not be the case.
It took a while to get used to, especially considering the way to learn how something works seems to always be to go and read source code on GitHub. There are good things and bad things about this approach but I fear it makes it difficult for your average user to use since it is a bit of a barrier to entry.
I loved how extensible the system was, and how easy it was to make modifications simply by modifying configuration.nix. It made it really feel as though my system was truly 'mine', in some ways being similar to the feeling I get from setting up an Arch machine. Being able to just recreate a machine by cloning your configuration onto the new machine, and running 'nixos-rebuild switch' is extremely powerful. Being able to roll back machine is also amazing. Something else that really impressed me was how great though ZFS support was. Setting up a ZFS system on Nix is one of the best experiences I've had using ZFS on Linux. It really feels as though ZFS is a first-class citizen on Nix, not something tacked on top. It's possible to set up ZFS without adding any sort of external repository like with Arch.
I ended up leaving NixOS after running into an issue that I would get related to UEFI, and at that point didn't have the time to track down the problem, but I'm really excited for where NixOS is going in the future, especially when declarative user environments https://github.com/NixOS/nixpkgs/pull/9250 (user configuration) with NixUP becomes a thing. I will definitely be coming back to it in the future.
I'm not sure how Gobolinux organizes files exactly, but with NixOS, the hash in /nix/store/<hash>-packagename is produced by combining every input that might influence the package. By input I mean for example:
- The source: git revision / source url / etc.
- All dependencies, which means if a dependency of a dependency of this package changes, it gets a new hash
- Build instructions, including build flags
This wouldn't be possible with a more conventional file structure.
https://github.com/pauldotknopf/darch
It is essentially Docker, but you can boot bare-metal with it. It even uses Docker under the hood.
Every boot is a fresh boot. You can read-write during boot, but all changes are made to ram. You mount with fstab things you want persisted, like home.
---
Layering
----
I can change my inherited build without rebuilding everything.
For example: base > xorg-nvidia > plasma > homepc I can easily switch desktop environments by making "homepc" inherit "i3", or w/e.
Each layer is essentially a "Dockerfile". If you understand docker, you understand how the images are built and managed.
---
Fast switching.
---
I can create images for different dev environemnts/clients. I can create an image purely for gaming. I can create an image steam.
Darch integrates with grub, so entries will dynamically show up on boot.
---
Sharing with DockerHub.
---
Sharing with other machines (and other people).
Since they are docker images, I can easily upload them to DockerHub.
I can update my local build by doing "docker pull && darch extract && sudo darch stage".
I will soon setup Travis-CI to auto build and push to DockerHub. Then, I never have to build locally. A cron job will queue a build every few days to make my image "rolling". I can easily revert back to previously working docker images (all tagged by date).
- The really powerful NixOS module system, which let's you specify your system declaratively and rather simply: Which programs should be installed, what systemd services should run, use grub for booting, allow this ssh key to login to this user, enable the IPFS daemon, etc. All options (for the 17.09 release) are available here: https://nixos.org/nixos/options.html
- System generations: Upon changing an option in your system config, you run `nixos-rebuild switch` which will build, activate the new system and add it to the system generations. All of these are bootable from your bootloader! So if something breaks miserably, you can just rollback to a previous one that worked. New kernel breaks on your hardware? No problem, just roll back.
- If you have a new nixos machine, you don't need to run a bunch of odd commands to have it set up. Since the config is declarative, everything you set up is right there, and all you need is a `nixos-rebuild switch` to make it work.
- Wanna test some configuration but are too afraid to use your machine? Just build a VM with it by doing `nixos-rebuild build-vm`. Like with everything else with nix, this fetches all dependencies, no need to install them manually.
Even though containers have had a lot of buzz and tooling built around them in the past few years, it's good to see how other approaches can solve similar problems with different pros and cons.
I'm sure nix is cool but this sort of sounds like something we just end up using docker for these days (for better or worse).
It seems that Docker is more concerned with containment and Nix with packages. I would love to be able to use Nix to define my project+dependencies and Docker to run it.
I'd be surprised if someone hasn't already made a baseline NixOS docker image to build off of.
0. https://devmanual.gentoo.org/general-concepts/slotting/index...
It's got some clever fundamentals that allow for really cool (and clean) features because of it.
On the other hand, trying to use NixOS on my laptop has been struggle after struggle. Every update breaks something. Many times updates cause the entire system to crash. Different combinations of display managers, window managers, and various system level daemons interact in complex ways.
None of this is the fault of NixOS, really it is the fault of the Unix philosophy scaled to the level of the desktop Linux ecosystem, combined with the traditional assurance on upstream developers that packagers in distros like Ubuntu will fix their shit.
What would be really great for NixOS would be a set of various well tested base configurations for the various DE/WM combos, like all the spinoff Ubuntu distros. These would fix the versions of all the fragile graphical components on some kind of release schedule, while probably still using nixos-unstable underneath for all the relatively reliable stuff like the kernel, emacs, vim, coreutils, etc.
If you want to learn more about Nix, I suggest starting with this excellent article on InfoQ: https://www.infoq.com/articles/configuration-management-with...
I also gave a talk about why I think Nix is so great: https://vimeo.com/album/4676161/video/223525975
Finally, I highly recommend the PhD thesis that Nix is based on: https://nixos.org/~eelco/pubs/phd-thesis.pdf
https://medium.com/@Pinterest_Engineering/continuous-integra...
I install any PACKAGE of version VERSION in:
~/stow/pkgs/PACKAGE/VERSION/
~/stow/pkgs/nim/0.17.1/
~/stow/pkgs/nim/0.17.1/bin/
~/stow/pkgs/nim/0.17.1/lib/
~/stow/pkgs/nim/0.17.1/share/
~/stow/pkgs/PACKAGE/VERSION/{bin,lib,share,..}/..
~/stowed/{bin,lib,share,..}/..
I skip the "restow" step if I just want to "soft uninstall" a package i.e. remove it from PATH, PKG_CONFIG_PATH, MAN_PATH, .. by removing it only from the STOW_TARGET ~/stowed.
"Hard uninstall" would be simply:
\rm -rf ~/stow/pkgs/PACKAGE/VERSION_I_DONT_WANT/
\rm -rf ~/stow/pkgs/PACKAGE_I_DONT_WANT/Nice post, and I'm glad you're enjoying NixOS!
I'm a NixOS contributor/committer, and the principle author (and co-maintainer) of the Bundler-based packaging integration. If you or anyone reading this has any feedback and/or questions about NixOS and Ruby (or NixOS in general) feel free to shoot me a message. I idle on #nixos on Freenode (as cstrahan), and you can email me at charles {{at}} cstrahan.com.
Cheers!
-Charles Strahan (cstrahan)
This is treating the symptom instead of the root cause, that being ineptitude to design backward-compatible software. We as an industry should fight such band aids and take a stand. Yes, designing backwards compatible software is hard, but exactly that is the difference between engineering and amateurism. And if one is getting paid to program, I’m going so far as to argue that it’s one’s job to have their head hurt solving such difficult problems so that users of computers don’t have to.