Writing a Package Manager (opens in new tab)

> Apple already uses something like this today

Some Linux distros too have things like this but unfortunately there is no buy-in across the ecosystem so "sandboxing" is done in a half-baked way.

The problem is when applications in general aren't written with sandboxing in mind, and when you have to choose between apps not working properly or having a leaky sandbox, you will opt for the latter.

I wish some big corp bit the bullet and ported hundreds of apps to a new, sandboxed environment in Linux, while attempting to upstream the whole effort. This would necessarily involve things like fully migrating to Wayland (X11 security is awful), only granting filesystem access through distro-sanctioned file pickers (so you need some coordination among Gtk, Qt, and other toolkits), and generally having a deny-by-default policy: first you make secure, then you fix what broke.

This idea is as confused as people who claim that they "don't need package manager because they have Docker".

So what if you will have containers / jails? -- You still need to install multiple components into the same container / jail... because you need them to work together. It's not solving the problem at all. Of course, containers are useful, but not for the purpose of solving installation of software comprised of multiple components problem.

Isn't this already how Android handles app permissions? Each application runs as its own user for the sake of security. The Application Sandbox is a pretty cool interpretation of the long existing Unix user/group paradigm.

How do you build the app with its dependencies without a package manager?

Like silverblue with toolbox?

The closest thing we have at the moment is conan[1]. It’s a cross platform package manager that attempts to implement “integrations”, whereby different build systems can consume the packages[2]. This is a big problem with package management in C/C++, there’s no single, standardised build system that most projects use. There isn’t even a standardised compiler! So when hosting your own packages using Conan, often you need to make sure you build your application for three different compilers, for three different platforms. Sometimes (for modern MacOS) also for two different architectures each.

If you control the compiler AND build system you can get away with just one package for most cases. This true for Microsoft’s C/C++ package manager, NuGet[3]

Historically, the convention has been to use the package manager of the underlying system to install packages, as there are so many different build configurations to worry about when packaging the libraries. The other advantage of using the system package manager is that dependencies (shared libraries) that are common can be shared between many applications, saving space.

[1] https://conan.io/

[2] https://docs.conan.io/1/creating_packages/toolchains.html

[3] https://devblogs.microsoft.com/cppblog/nuget-for-c/

FYI pip is specifically not a package manager, it's a package installer.

Pip does not attempt to resolve dependency conflicts of already installed packages, only the ones it is currently trying to install. Nor does it try to manage the life cycle of installed packages, such as removing transitive dependencies never specified or no longer needed or create consistent environments with a lock file.

As package specifications have become better defined (rather than "whatever setup.py does") and are being better enforced (e.g. Pip will soon reject version numbers that are non-compliant) there are several attempts at writing full-fledged package managers (Rye, Huak, Pixi, etc.) and I'm sure once one of them gains critical mass it will replace Pip quickly.

There isn't any. There are a few partial ones like conan, etc.

And many people in the community are against it, you'll hear stuff ranging from "just use the distro package manager" to "don't use any, make it a single file library".

Personally, I'd say all hope is lost.

Use Bazel.

A couple of reasons:

1) I wanted human readable specs and lockfiles.

2) I didn't want to introduce a dependency (working with SQLite in Go is pretty ugly, it either requires GCC or a ton of other dependencies for a pure Go implementation).

SQL not so good for tree structures. Recursive CTEs are pretty gnarly.

Graphdb would be ideal.

local system log "files" and package databases would be an excellent example of using sqlite

Great point

There is quite a difference between "the author added a lockfile without understanding why they exist" and "I really don't see why they added a lockfile".

It's better to start with the latter if you seek to understand, not attack.

The lockfile in this post actually seems a bit more like a manifest, or at least that it's trying to do both things at once.

I'd expect you'd have human readable file to list your dependencies and versions in a simplistic way, and then a separate machine readable file that locks the versions for reproducibility. As it stands, the example in the article does not look pleasant to write by hand so you'd have to script something yourself.

A version number is just a label, and labels are mutable. A lock-file containing hashes will always resolve to the same packages (or fail).

> I've typically bypassed the need for a lockfile by simply checking in the dependencies. Dependencies belong in version control!

These are binary dependencies. You're checking shared object files into source control? Note that he's also doing this to add functionality to a binary installation of sqlite, which is presumably running somewhere like /usr/bin/sqlite. This isn't a custom application he's developing. The extensions are in his home directory. "Checking them into version control" would entail making his entire filesystem a Git repo. If you're willing to pay whatever IBM charges these days for ClearCase, something like that might actually work. Regular source control like Git, though? I don't think so.

> Environment variables too. Kill them all with fire.

I feel this way about host file entries.

It is shocking how common they remain, and not just in QA/Dev.

It totally works for Go with MVS, which bumps a lot of versions behind the scenes for you.

We don't, but it is the easiest way to ensure the package manager is adapted to the language and the library ecosystem. It is also easier to add features, remove misfeatures, and fix bugs when the package manager only has to handle a single language.

Once we understand package management better -- we don't understand it well yet -- it will likely be feasible to consolidate many package managers into a few omnimanagers.

In theory nix can solve that, in practice is more ergonomic to leverage a language's native package manager even in nix.

Need is the wrong word, sometimes it is necessary. Look at npm, long time they not include new features and made less progress. That time yarn was born. pnpm and other followed, cause npm did not have features, security solutions, and other things that people want.

In theory we need only one for all, but in reality, that is impossible.

These copy-pasted one-liners aren't useful. Skip them.

Imagine what'd we be using for package management today if package manager >= #2 author said "you know what, one already exists".

Nix solves that problem.

(While creating some new ones.)

You could argue that Bazel, Nix and Guix solve that problem but they introduce a lot of complexity as a result because it's not a simple abstraction. Hell, you could probably pull it off somehow in Cmake if you placed no value on your sanity. Most standard lib package managers double up as build tools for bundling your code after all.

It's a totally different problem space than the one asdf aims to solve, which is just managing multiple independent versions of some programming language binaries in a consistent way, so you're not dancing between nvm, rvm/rbenv/chruby, virtualenv, etc. etc.

Something like Meta Package Manager? https://github.com/kdeldycke/meta-package-manager

I could be wrong here, but given what sqlite's documentation includes and the fact his spec files don't even include dependencies, I believe sqlite extensions don't depend upon anything except sqlite itself (he says here "most" don't).

You'd need to ensure the extensions and sqlite were both compiled against the same libc, but I don't see what this package manager can do about this given the metadata doesn't seem to be available in these Github releases. In fact, going to the repo for the example in his spec file, the reason for the release was to downgrade the build host OS from Ubuntu 22.04 to 20.04 to resolve an issue with some user not being able to run this because of a missing GLIBC symbol.

This is an underappreciated problem and why actual distros include everything. If you're going to distribute binary files, you need to ensure they're compatible in many more ways than just the architecture and kernel matching. A complete package manager would include a build system and public registry with builds matching a complete machine triplet (as in, what you'd get by running gcc -dumpmachine). If you consume upstream Github releases as this is doing, they may not have that level of granularity, and in fact, we can see they do not.

Doesn't the article say that it doesn't deal with dependencies at all?

Precisely, I wouldn't call this a "package manager", it's a "package installer" (same as pip). Managing implies some form of interaction after components are installed (esp. ensuring that all installed components can work together). This project doesn't seem to manage anything, it's more of a fire-and-forget kind of thing.

Agreed. Version resolution is the interesting problem.

Most package managers use a SAT solver to resolve dependencies. The Dart team has a detailed write up on their SAT-based approach which is worth a read [1]. For contrast, Russ Cox presents an algorithm that doesn't use a SAT solver (intended for Go) [2].

[1] https://github.com/dart-lang/pub/blob/master/doc/solver.md

[2] https://research.swtch.com/vgo-mvs

Probably the simplest version of a "package manager" are git submodules. Pointing submodule to `master` is effectively the same thing as pointing a "real" package manager to `latest`. This is trunk based development, however you implement it.

One could easily argue that floating versions are considered harmful for releases outside of development team and should always be pinned, but on the other hand it is hard to argue against support for floating versions in development. As much as I dislike floating versions, I am not aware of any other way to force changes downstream.

Not running arbitrary code is a feature, that yarn brings back to the js ecosystem.

It sounds nice in theory, but it impacts the soundness of the whole system.

Maven didn’t support transitive dependencies from the beginning.

Most package managers work with version control already, they are not solving the same problems. Package managers deal with the building and dependency graph along with delivery of working executables. Version control solves zero of these problems.

If you haven't already researched them, you may want to investigate Plan9 and Erlang, both early systems embodying a distributed and networked philosophy of computing.

SQLite is quite good at staying backwards compatible, and this includes the APIs exposed to extensions, but extensions could definitely have a minimum SQLite version.

What do you mean by OCI and KCL?