https://guix.gnu.org/en/packages/python-pytorch-1.12.0/
This makes me wonder if Guix is ready for the real world.
ML and GPUs are a smallish subset of the real world, and a subset that's particularly plagued by proprietary software (especially the GPU side); I don't think it's a good litmus test for "is guix useful for real work".
That firmware could literally ship on the card totally invisibly and that would be fine for guix, but because it can be patched in the field, no.
I don't get it.
I think Guix is more than ready for providing development environments, even for proprietary apps on open source technology stacks (but Nix will probably be easier to pitch). As an operating system much less so.
If you do know the build options to make this work, Guix will gladly consider your patch.
The in-progress diff would be,
diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index e702e499fc..73af29487d 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -101,6 +101,7 @@ (define-module (gnu packages machine-learning)
#:use-module (gnu packages swig)
#:use-module (gnu packages tls)
#:use-module (gnu packages video)
+ #:use-module (gnu packages vulkan)
#:use-module (gnu packages web)
#:use-module (gnu packages xml)
#:use-module (gnu packages xorg)
@@ -2897,6 +2898,10 @@ (define-public python-pytorch
(build-system python-build-system)
(arguments
'(#:phases (modify-phases %standard-phases
+ (add-before 'build 'use-vulkan
+ (lambda _
+ (setenv "USE_VULKAN" "1")
+ (setenv "USE_VULKAN_SHADERC_RUNTIME" "1")))
(add-before 'build 'use-system-libraries
(lambda\* (#:key outputs #:allow-other-keys)
;; Tell 'setup.py' to let 'CMakeLists.txt' know that we
@@ -2973,7 +2978,11 @@ (define-public python-pytorch
pthreadpool
protobuf
pybind11
+ shaderc
sleef
+ glslang spirv-headers ;spirv-tools ; not sure why this is needed
+ vulkan-headers
+ vulkan-loader
xnnpack
zstd))
(propagated-inputs
FAILED: bin/scalar_test ld: /tmp/guix-build-python-pytorch-1.12.0.drv-0/source/build/lib/libtorch_cpu.so: undefined reference to `glslang::TShader::setNanMinMaxClamp(bool)'
ld: /tmp/guix-build-python-pytorch-1.12.0.drv-0/source/build/lib/libtorch_cpu.so: undefined reference to `spvtools::CreateCompactIdsPass()'
[...]
So that's why it's not enabled.
There are actually much less rules than "normal" programming languages. I had started to write my own basic functions that would be in a "standard library" (trim(), trunc(), remove() etc) and got the hang of it after a couple hours.
The only thing I don't like about the language for "serious" work on massive projects is missing static types (may make it less readable though..), and the tooling (most people lean on Emacs) could be better.
So I'm guessing there's some additional conventions for function naming, besides appending ? to the name for boolean return and ! for functions that mutate.
[1]https://spritely.institute/static/papers/scheme-primer.html
It can't be a yaml file because it's essentially a program, it's a lot easier to write such things as a program.
Keeping this in mind, I find the Guile scheme syntax used by Guix quite intuitive and pleasant. It reduces the cognitive overhead to configure the system, at least for me.
Also each configuration file in Linux has already has their differences and quirks
- the module paths a bit weird (I’d find slash or dot separated more obvious)
- some values needed to be constructed with a call to list
- weird dsl for building was mixed in there
- it’s probably not obvious what should be string/symbol/whatever
I’m not sure anything should change though. It still feels a bit weird to me that it’s a program generating the package rather than a more direct ‘just data’ format.
I do actually prefer s-expressions to JSON, but only when used as pure data. Once you turn it into scripting language it gets ugly fast.
Tongue in cheek :)
One thing I couldn’t work out how to do in guix was to go the whole way through. Something like:
1. Install / use some program
2. Find some bug or potential improvement
3. Run the command to be dumped into a dev environment for that
4. Fix or investigate the bug; or implement the improvement
5. (The step I don’t know how to do) Use your fixed/improved version in your operating system
6. (Also don’t know how to leverage guix for this) Share your improvements with others
It feels like 5 is somewhat at odds with how guix is meant to work as step 4 is so mutation-heavy, but it also feels like something that an OS which wants to be as gnu as possible should really want to support. And gui should be able to make it safer too by giving you rollbacks. Maybe there’s an easy way to do it and I just don’t know it?
guix install foo --with-source=foo=yourfixedversion_src.tar.gz
Or:
guix install foo --with-patch=foo=yourpatch.patch
(note: usually packages are installed as regular user into your user profile)
See also section "Package Transformation Options" in "info guix".
For step 6, send a mail with the patch to the package definition (in an existing scm file in https://git.savannah.gnu.org/git/guix.git ) to the guix-devel mailing list. But whatever you did manually to get yourfixedversion_src.tar.gz you now should automate by programming the patching process in Scheme instead--editing a guix checkout (usually just to add: (add-after 'unpack 'patch-problematic-stuff-4711 (lambda _ (substitute* "somefile" (("regex1") "replacement"))))). For more complicated fixes, add a patch to the guix checkout and make Guix use it in Scheme (and contribute it to the actual foo upstream project--which we usually do anyway).
See section "Contributing" in "info guix".
Further automation welcome-- and probably not that difficult to add. Just keep in mind that each package build is in its own container (it's kinda like Docker would be)--so no mixing text editor and building and weird user-defined pauses (for example there is no text editor in the build container).
This is exactly right, and I think it's one of the coolest things about Guix. Guix System (the full OS based on Guix) seems almost like the GNU that was dreamt of before Linux. It even has an option built on the Hurd kernel, and it uses an init system and initrd images based on Guile Scheme, which is both the language used for Guix itself and as the extension language for GNU projects.
Guix definitely advances the whole
> Learn one simple, high-level language and you can hack on every aspect of your system
idea further than we've seen on any modern/contemporary system.
> Aside: I think this is also great about Emacs; you can easily jump to the source of most things and there’s usually a way to hack or modify or inspect or debug them immediately
I think Emacs is definitely an inspiration, or a model of the virtues intended.
With guile, you get none of that, there's seemingly no debugging tooling at all. The best Emacs integration you get is Geiser which is not a GNU project and doesn't offer the debugging functionality mentioned above.
This is very surprising considering that Emacs and Guix are under one roof, and Guix is arguably GNU's flagship new software. Why do the development tools suck?
There is no way to build the package graph on macos. There is no glibc port for macos, and there's no free toolchain we could legally use.
There is little value in building Guix when we are forced to use a different proprietary toolchain, a different C library, and throw away the from-source bootstrap in exchange for Gigabytes of proprietary blobs. You might as well use Guix System in a virtual machine on macos.
As far as the ideological reasons goes, last time I looked for this, someone on a mailing list asked how you'd go about porting guix to macOS and some core team member answered they had no interest in supporting a non-FOSS system.
Why does gnu-build-system use %standard-phases (a symbol with %)? You'd think it could just use standard-phases. What is the big namespace concern. The % prefix is a Lisp convention for implementation internal symbols, not to be used by user code. Kind of like double underscore in C.
It could simply be the mplicit default. If most rules use %standard-phases, it's silly to be repeating it all over the place.
modify-phases could have a syntax like:
(modify-phases {phases-object | clause }*)
(modify-phases
(add-after 'unpack 'bootstrap
(lambda _ (invoke "sh" "bootstrap")))
About the lambda, a simple idea would be that if the registered function is a list object (pair), then it's treated as list of exec arguments:
(modify-phases
(add-after 'unpack 'bootstrap '("sh" "bootstrap")))
A macro like (cmdf "sh bootstrap") could do the parsing and generate the lambda + invoke. The "f" reminds you that it doesn't run the command but returns a function.
> Why does gnu-build-system use %standard-phases (a symbol with %)
AFAIK %symbol indicates a constant in Scheme.
It's unfortunate that it isn't written in Common Lisp. CLOS would've been immensely useful there instead of the ad-hoc object system via Scheme records.
AFAIU, asdf is for providing particular versions of tools (e.g. python, etc.). In the original post, guix is also used to provide library dependencies.
Guix is the free software oriented, lisp flavored cousin of Nix. -- I've used Nix. It's got enough rough edges that writing nix code is pretty much only accessible to enthusiasts.. though, those who can get past the rough edges do recommend it.
The added complexity of guix/nix allow for having an OS distribution where the whole OS config is declared/reproducible.
The documentation is good
It still has problems here and there, and packages missing. Probably due to the lack of wide adoption for being new and different
But packages that are missing, most are easy to create or find online, to use in your private repository, or submit to main repository
There was less difficulties than expected. Things that didn't work, or I didn't wanted to waste time fixing