undefined | Better HN

0 pointstgamblin10y ago0 comments

Got to agree with davexunit here. Language package managers aren't going to handle your non-<language> dependencies. One place where this comes up in spades is with numerical software. Example: frequently you want to build numpy/scipy with fast math libraries (BLAS, mkl, etc.), which might be in C, Fortran or whatever. Python's myriad custom package managers have no support for that level of customization so you're back to manual installs.

Spack [1] attempts to solve this by allowing extensions [2,3] for languages that have their own module systems. Basically you can embed language-specific module enablement logic in the package, and you can enable/disable packages in particular Python installs. You can then build several versions of numpy with different dependency libraries and swap them in and out. It's a compromise between language and system package managers. Right now it's too stateful in that the modules are linked/unlinked into a python installation. We have plans to provide something more like virtualenv or conda environments.

Functional package managers are nice, as are reproducible builds, but they come up short for customizability. I'd have to write an awful lot of packages with Nix and Guix to try out many different versions, compilers, dependencies, etc. Spack has a syntax [4] that allows these to be composed arbitrarily. It's not fully reproducible right now (though it could be if we went down to glibc and used a chroot jail), but the packages are much more composable than the current crop of functional package managers.

[1] http://github.com/llnl/spack/

[2] http://software.llnl.gov/spack/basic_usage.html#extensions-p...

[3] http://software.llnl.gov/spack/packaging_guide.html#extensio...

[4] http://software.llnl.gov/spack/basic_usage.html#specs-depend...

0 comments

2 comments · 1 top-level

rekado10y ago· 1 in thread

> Functional package managers are nice, as are reproducible builds, but they come up short for customizability. I'd have to write an awful lot of packages with Nix and Guix to try out many different versions, compilers, dependencies, etc.

I disagree. At least with Guix you have programmatic access to the dependency graph (since package objects are just Scheme values). You can traverse the package graph and easily swap out all occurrences of a particular package. You do not have to write the package expressions yourself as you can just modify the package objects directly.

See also the paper "Reproducible and User-Controlled Software Environments in HPC with Guix"[1].

Since the paper was published Guix also gained command-line syntax to rewrite the dependency graph on the fly, e.g.

    guix build guix --with-input=guile=guile-next

which maps the "guile" input to "guile-next" when building the "guix" package. Recent versions contain a pretty generic package transformation framework.

[1] https://hal.inria.fr/hal-01161771/en

tgamblinOP10y ago

In Spack this would look like:

    spack install guis ^guile@<new version>

How do you handle optional dependencies in this model? Or, a more complex case, dependencies that are only present for certain versions of the package? Look at the gcc package in Spack:

    https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/gcc/package.py#L54

This builds gcc 4.x and 5.x versions, and it's pretty concise. The `mpc` and `isl` dependencies are only present when the version of gcc is 4.5 or higher, or 5.0 or higher, respectively.

There's an iterative DAG-building step called concretization that looks at these types of constraints and tries to solve for a satisfactory DAG, which is then built in a functional style. I don't see how I would model that in Guix, and but in Spack it's a few lines, and there's command-line syntax to customize versions on the command line, not just programmatically. e.g.:

    spack install gcc@5.0 ^isl@0.11.1 ^mpc@1.0.2

j / k navigate · click thread line to collapse