pip, for instance, eagerly resolves the highest version of a dependency first found in the tree and doesn't backtrack AFAIK [1]. Cargo sounds as though it behaves similarly, but will actually backtrack [2]. I've also seen package managers like 0install that will use SAT solvers to solve dependency constraints [3].
I've found lack of backtracking on conflicts to be frustrating, but I'm curious how the other two (and other conflict resolving strategies) compare in practice.
[1]: https://github.com/pypa/pip/issues/988
https://github.com/rust-lang/cargo/issues/2064
and graydon wrote a POC of using the Z3 solver to find a solution where semver constraints were satisfied, optimizing for the latest versions within those constraints:
https://github.com/graydon/z3-rs/blob/master/tests/semver_te...
Glide has a dumb solver in that it goes with the first best version it finds. It will tell you if there is a later conflict that you can manually resolve.
There is a package, discussed here called gps, that is being prepped to merge into Glide. This solver has a back-off strategy. It's loosely based on the solver in dart. The idea is to find the latest newest version that works for the complete tree.
We've been incrementally working to get a good solver and are almost there.
1. RubyGems is the package manager. Bundler is a meta-layer on top of that which does full dependency resolution to find the right version before installing. Bundler builds on the capabilities provided by RubyGems (and will be integrated into RubyGems in the future).
2. Bundler does support vendoring. It is widely discouraged, but I take advantage of it in my app packager for deploys at work (Cartage: https://github.com/KineticCafe/cartage, specifically cartage-bundler https://github.com/KineticCafe/cartage-bundler).
3. RubyGems and Bundler can use alternative sources (my work applications use both public and private sources simultaneously).
One thing I don’t see that would love to see tracked and solved is authenticity verification. RubyGems has support for signed gems but it’s not widely used and hasn’t really been validated as Correct.
They should probably also look at CocoaPods and Carthage (iOS and macOS build dependencies) and whatever Swift provides.
It is also widely encouraged. Some engineers:
1. Don't trust dependencies to always be there.
2. Write apps that need to work in disconnected environments -- ie with no internet connectivity.
I work on the Cloud Foundry buildpacks team for Pivotal. Being able to stage and launch apps in a disconnected environment is A Big Deal for a lot of companies.
Heroku mostly does this right with its 'slug' system, and I made Cartage specifically to be able to make deployable packages that use dependencies. You have a build machine that’s connected to the network and can reach RubyGems (and, in the future with some plug-in work, other packaging systems—I expect to add Node, Elixir, and Lua soon) and that will vendor the resulting build and give you a tarball that is just installable with all of the dependencies of your package.
Using a new language is no longer a matter of cracking open a book, installing a compiler, and firing up vim. You've got to change your whole lifestyle now.
CPAN, RubyGems, etc. all exist because system packaging systems are (a) specific to a particular distribution, (b) generally many versions behind the development versions, (c) really hard to get into unless you are already a system packager for a particular distribution, and (d) effectively don’t exist for two of the largest platforms where you will find users.
I’d suggest making some sort of meta-system packaging system that works for all the languages, but xkcd 927 (and I wouldn’t use it myself, anyway, because it would probably not install Ruby packages correctly and would solve things worse than NPM already does).
- It does mean that the common libraries are installed once - conflicts between common libraries are more transparent - System tools make sure that package stays up-to date
* building on not-linux -- you know, where almost all personal computers exist. Gee, wouldn't it be nice if most computers could actually compile your code?
* Having only one build config, only having to maintain one publication (ugh need to maintain deb, rpm, nix, chocolatey, brew, whatever the hell arch and suse use...)
* reproducible builds -- distros are well-known to randomly modify libs, making "foo x.y.z" in yum and apt basically random. It's not even clear to me I'll get the same thing from apt on different days!
* Iterating and collaborating on fledgling projects which rightly wouldn't be accepted into a distro -- or just iterating on a timescale smaller than "months"
* local deps -- using system-global deps (as preferred by most linux package managers) is known to be a huge nightmare for any language package manager that's tried. Hence pyvenv. Hence why so many have thrown up their hands in disgust and just started dockering all the things.
(inb4 someone comes to explain that Nix or whatever probably magically solves all these problems)
Like all things in languages, the world needs to be reimplemented. Which is too bad, but I also never see a great alternative. Eventually people want everything they work on to be in the same language, which of course has the advantage that anyone in that language can hack on it, but the disadvantage that we always start anew every time a language gets created.
Containers don't "solve" everything, but some of the primitives used to build them can be used to severely restrict process and user privileges -- to the level that it's much more secure than a "standard" unix-y shared environment.
I wish devs would just tell me the libraries I need, and I'll get the packages myself.
Unless your on the bleeding edge of development, you don't need the newest versions of a lib. And you probably shouldn't be developing on the bleeding edge anyway. There's a reason apt-get is so stable, and is used everywhere.
The caveat to all that is: I hate making packages for apt-get. It's a terrible process, and I completely understand why so many devs want to avoid it.
This is a decent overview on technical details: http://vincent.bernat.im/en/blog/2016-pragmatic-debian-packa...
[#] I assume you meant the friction with the tools, not the fact that package builder needs to chase down all the dependencies and generally prepare many source tarballs to be included in a repository.
I've seen this argument come up a few times now, and I have a hard time reconciling this rosy view of system package managers at the center of a development process with my own experiences. That process was never as easy as "cracking open a book, installing a compiler, and firing up vim" and usually entailed a fair amount of trial and error, environment variable fiddling, and sadness from having to chase down a derelict PPA (or the equivalent) hosting the actual dependency version I need. Then more sadness as the number of projects I work on increases and my dependency versions bifurcate.
Some peers of mine recently experimented with building a Nix buildpack and concluded that it won't work. Nix has hardcoded paths for its core directories, and won't work in a security-constrained container.
Apt is no more standard than pip, gem, or any of the others.
Try maintaining packages for Debian, Red Hat, Arch, Gentoo, OSX, Windows, and a dozen others, for a while, and see how quickly you change your mind.
> You've got to change your whole lifestyle now.
Oh please.
https://www.dartlang.org/tools/pub
It hews pretty closely to Bundler's model (versions, version ranges, lockfiles, shared dependencies), though. So I don't know if it gives you much insight beyond "Yes, another language team things Bundler is on the right track."
Personally, I have a hypothesis that states that all programming languages either have a package manager that works like Bundler or will end up getting one.
https://metacpan.org/diff/file?target=ETHER%2FURI-1.71%2F&so...
This information is also available through an API for integration into command line tools.
In fact, the original creator of Glide (Go package manager) wrote about Perl and CPAN when talking about Go at http://technosophos.com/2015/09/02/dont-let-go-be-condemned-....
Other things from the Perl ecosystem that should be copied are:
- CPAN Testers which automatically tests every package on multiple systems from Windows to Solaris. This helps identify portability, backcompat, and regression issues.
- CPAN mirrors which ensure that there isn't a single point of failure. This might not be as important now with fast networks and high uptimes, but it also ensures that everyone can replicate all of the code at anytime. I believe R's CRAN does this.
All modern packagers provide now such src urls, esp. go went ahead with this idea.
Fwiw pip isn't even able to enforce versions correctly (packages are installed as the file is read, and can conflict with previously expressed constraints). Or report installed versions correctly (it's possible that packages are half-installed or installed but not reported as such by pip commands).
$ pip freeze -r requirements.txt > constraints.txt
$ pip install -r requirements.txt -c constraints.txtWe were sad.
* All current package managers are either language or OS specific. What if you have an application with code written in multiple languages?
* NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.
* Some package managers do better than others with regards to managing packages. I found NPM encourages dependency hell and very little management tools for dependent or installed packages.
* A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.
I am trying to work on a solution to these problems at https://github.com/prettydiff/biddle
guix and nix both work cross-language and cross-distro. Still OS specific though since only linux AFAIK. Also, containers partially solve this problem.
> * NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.
Any package manager that doesn't do integrity checks is a bad package manager. The only one I know of currently that doesn't is npm, but I haven't looked deeply into every available package manager.
> * A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.
What package managers don't let you self host? I'm truthfully not aware of any. Even NPM does according to a quick google.
> I am trying to work on a solution to these problems at https://github.com/prettydiff/biddle
From your readme: "biddle is inspired by the incredible awesomeness of NPM". Since NPM is literally the worst package manager I have ever used, that line makes me want to stop reading and never touch biddle. I'd word it differently.
Edit: Reading biddle further. Dependency management and central hosting are some of the primary reasons to have a package manager. At least for me, that kills any interest at all. I imagine there's a niche market though?
(Aside from the lack of package integrity check, which I'll grant, sucks).
It's already very hard to find a programmer that writes a lot in more than one language and cares about packaging/deploying software.
I'm not arguing the point just trying to understand the decision.
At least, that's my understanding of what they meant.
Probably because of this[0]:
> Note that Cargo does not yet support vendoring in a first-class fashion, but this is a hotly desired feature and coming soon!
[0] http://doc.crates.io/faq.html#how-can-cargo-work-offline