Python Pip 20.3 Released with new resolver (opens in new tab)

I will give it 5-10 years. If they are serious about it, it will last that long. They're about halfway there.

Package management is terrible work. Nobody appreciates it. It's extremely complex, it has to work in enormous numbers of configurations, and very minor errors can have catastrophic impact to security or availability of systems.

How do you deal with keeping your top level dependencies and exact versions of all of your dependencies of dependencies separate in a way that's sane and 100% reproduceable for a typical web app / repo that might not in itself be a Python package?

I'm in the same boat as you in that I'd like to keep using pip but the lack of a lock file is very dangerous because it doesn't guarantee reproduceable builds (even if you use Docker).

In Ruby, Elixir and Node the official package managers have the idea of a lock file. That is the only reason I ever look into maybe switching away from pip.

Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies.

I've been bitten by issues like this so many times in the past with Python where I forgot to define and pin some inner dependency of a tool. Like werkzeug when using Flask. Or a recent issue with Celery 4.3.0 where they forgot to version lock a dependency of their own and suddenly builds that worked one day started to break the next day. These sets of problems go away with a lock file.

If you have a working setup and especially a strategy for updating routinely (e.g. using pip-tools) and pinning dependencies with hashes, there’s less reason to change. The biggest reason I’d give is consistency and ease of adoption, which might be coming down to “do you spend much time on-boarding new developers?” or “are you supporting an open source community where this causes friction?”. If you aren’t spending much time on it, perhaps try it on new projects to see what people think in a low-friction situation - in my experience that’s basically been “I like spending less time on tool support”.

They either haven't experience the pain or are oblivious to it. Pip old resolver is borked[0]:

> [The new resolver] will reduce inconsistency: it will no longer install a combination of packages that is mutually inconsistent. At the moment, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, right now, pip install "six<1.12" "virtualenv==20.0.2" does the wrong thing, “successfully” installing six==1.11, even though virtualenv==20.0.2 requires six>=1.12.0,<2 (defined here). The new resolver would, instead, outright reject installing anything if it got that input.

[0]: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...

I was one of those setup.py + requirements.txt (generated by pip-compile).

Though, poetry is actually quite good. There are still some things that I wish it had, like plugin support (for example I really miss setuptools_scm) or being able to use it for C packages.

But if your code is pure python it is great from my experience. The dependency resolver is especially good.

beware of zealot geeks bearing gifts. if your environment is currently working fine and you are only interested in running one version of python and perhaps experimenting with a later one then venv + pip is all you need, with some wrapper scripts as you say to make it ergonomic (to set the PYTHON* environment variables for your project, for example)

poetry has several advantages which I can no longer live without.

1. Packages are downloaded in parallel. This means dramatically quicker dependency resolution and download times.

2. Packages can be separated for development versions production environments.

3. Poetry only pins the packages you actually care about, unlike a pip freeze. One application I work on has 15 dependencies, which yields a little over 115 packages download. pip freeze makes it impossible to track actual dependencies, whereas poetry tracks my dependencies - and the non-pinned packages are in the poetry.lock file.

The rest is nice, but the above is essential.

I was happy with Pip until I spent time in the NPM/Yarn world. Frustrated with Pip I switched half of our projects to Pipenv. However, I found that it struggled to resolve dependencies. Poetry works like a dream now, and life is so much easy now that we have switched all our projects to it.

The methodology of specifying your core dependencies, but also having locked version of your dependency's dependencies works really well.

AND you can easily export to requirements.txt if you prefer to use that in production.

> I occasionally ask our principle / sr. Python engineers about this, and their response is always, "These things come and go, virtualenv/wrappers + pip + requirements.txt works fine - no need to look at anything else."

At the macro level, this seems like a bit of a self-fulfilling prophecy: if all the senior and principal engineers using Python don't care to take a look at something else, then it's not too surprising that new solutions don't end up sticking around. That isn't too say that there does necessarily need to be a change in the Python community's choice of package manager, but the rationale for not even considering looking at other options doesn't seem super compelling.

I personally rather avoid any per language package managers and opt for distro level package management (rpm/deb). Thats something that has bern there pretty much for ever & has all the hard & dirty depsolving issues long solved.

Also I can use it for all software regardless of what laguage it's written in, not to mention having to learn multiple half-baked package managers per laguage!

And lastly any non trivial software project will need dependencies outside of the world of its own package manager anyway, so why not go all in properly with rmp/deb & make everything easier for your users & your future self.

> Is there a reason to look at poetry if you've got the pip/virtualenv combination working fine?

You probably have a bunch of scripts that do what poetry does (either that, or you repeat the same commands over and over A LOT).

Switching to poetry might have some initial overhead, but a big upside is that you stop using custom, internal tooling, and use something industry-standard. Importantly, it makes it easier for you to understand external projects (since you're familiar with the standard tooling), and faster to onboard newcomers.

I’m curious how they ensure every developer is using the same versions of things, as well as how they manage dev dependencies and transitive dependencies.

pip + venv + requirements.txt doesn’t solve this out of the box while most languages have common tools that do. Either they’ve rolled their own way to manage these things, or they’re rolling the dice every time they deploy.

You're losing the ability to easily update using fuzzy specs and a lockfile if you're using pip vanilla without pip-tools or poetry.

I hope the second is never changed - it’s 2020, time to start authenticating the servers you’re downloading and executing code from

Internal doesn’t mean secure

Pip + setuptools is "it's working, no need to change" in the same way that "SVN works so why bother with Git?". It seems fine until you try the alternative, and then the thing it replaces just feels painful to use.

This is not my experience, it seems to be different for different dependencies being installed.

What complexity does Poetry have over pip? This is a serious question. Yes, you have to install it on top of your Python install, but other than that?

Particularly, it isn't an installation method someone writing a package manager should consider. Bootstrapping tends to be clumsy, but this is too much.

The exact same thing happens behind the scenes when you pip install something. (Download source from a trusted website, run with interpreter)

`pipx install poetry` works just as well

Actionable: use backward-compatible behavior by default. Use opt-in for new features. Question: is the new resolver opt-in or is it enabled by default in the new pip version?

The action is simple, stop making breaking changes. There is frustration with breaking changes being in the history, but that is much less troubling than the reoccuring answer to the question “What is wrong with my code?” being “A new version of the package manager was released.”

It is also very possible to write code to completely work around old issues so the newest version of your package manager just works or at least outputs clear error messages with mitigation steps, but instead what you get is obscure stack traces when you run pip install.

As long as you don't have lots of C or Cpp dependencies, Python packaging is apparently totally fine.

Unfortunately I don't live in that world :(

I take it you're doing a lot of greenfield projects with the latest versions of Python then. I've encountered issues with Setuptools as recently as ~3 months ago:

https://github.com/pypa/setuptools/issues/2352

I've also had to play the pip version pinning game a few times in the past.

Here is one from September:

https://news.ycombinator.com/item?id=24336058

Same here.

I don't know Rust well enough to comment, but doing a lot of development in both JS and Python, JS is no cakewalk either. I'd say I deal with at least as many packaging/versioning issues on JS as I do on Python.

Another issue is the two mutually incompatible ecosystems of async vs. non-async Python libraries and frameworks. You now have these entire mirror universes of code, with "[X]-async" versions of existing synchronous libraries, and new async frameworks which require using these libraries. The years of work that've gone into non-async libraries are now basically all for naught, with lots of people reinventing the wheel for every possible thing that interacts with a network. (I'm also not a fan of async/await in general.)

I'm a big Python fan; it was my first language and still by far my most-used and the one I know best. But the "developer experience" is undoubtedly a giant mess compared to something like Go or Rust. (Rust also appears to be having some growing pains related to async, but everything else seems way more solid.)

Despite Python's catchphrase "there should be one, and preferably only one, way to do it", Go really embodies this across every dimension. One way to install the runtime, one way to format, one way to package, one way to do concurrency, one way to test, one way to write most things. And every version is almost completely backwards compatible.

It's kind of crazy that a language that (justifiably) prides itself in its programming simplicity also makes you deal with tangled messes like https://xkcd.com/1987/

My one solace is that some all-in-one third-party tooling is finally close to the level of convenience you get with Rust. pyenv and poetry save a ton of headache when it comes to managing Python versions and packages, and "just work" in a pretty similar way as rustup and cargo.

The JS ecosystem has just as many "issues" if you look at it objectively. Don't mean to be funny or lazy, but one need only look at this SO answer to see how weird/fragmented the JS package system is (the answer has a Table of Contents).

https://stackoverflow.com/a/39825582/1140382

I’m fairly certain that yum and apt meet these requirements for “traditional” package managers, I know Maven does as well.

I think it depends a lot upon the culture of the ecosystem, maven being fairly conservative is a natural consequence of Java being very enterprise focused.

but python has been around for a while, I feel the frustration comes from the fact that other platforms seem to have managed to get a single working tool much earlier.

In my experience, python dependency management + library installations + code sharing is way easier than the following:

C#, C++ (Conan) and Java, Linux(apt, yum, etc) and most certainly JS/TS.

What is so hard about "installing code, packages" in python for you?

I remember composer as the package manager that required a gigabyte of RAM to install a few dozen of dependencies for some framework.

Insert "Poorly Hacked Perl" joke here ?? /s ohh - lets do vi vs emacs next! :-)

heh. the down votes are so pefect.

For Postgres, there is also disabling fsync delays (20x speed claimed) and even running it all in memory.

https://dev.to/thejessleigh/speed-up-your-postgresql-unit-te...

I'd bet its becoming increasingly popular/convenient to do so with the growth of container-based test suites - really trivial to include a "postgres-ram" image for your test stack instead of one that uses disk, and you don't even need to know what tmpfs is to do so.

These days, “high amounts” is a lot more than most projects need - even ultra-portables have 8GB or more. Most projects don’t have unit test suites limited on file I/O measured in gigabytes unless they also have a budget.

Where do you see anybody thinking that "came from using containers or the cloud"?

> 20 versions are no joke.

Versions ≥ 18 refer to the years 2018+; they don't refer to the number of versions released anymore.

It looks version 21 (!) will remove support for Python 2.7 - now that should be interesting...

Wait until you hear about Chrome!

In all seriousness, the constant warnings mean that users may ignore serious upgrades.

You may need to see an expert if a little warning message actually gets you that angry.

That and taking 5 minutes to figure out that it can't possibly do what you told it to (except, of course, it can).

Look at the pip issue history.

most recently (october?) they vendored some package that was a system dependency before, and it broke the vendored versions of pip (eg. on debian).

I get, “not their fault” debian goes and modifies packages... but from a user perspective: it broke.

I would say my experience is roughly on every six months something to do with pip breaks for me... but I really cant be bothered trying to keep track of it.

I just try to avoid using it now. Down vote all you like, I don't care. Pip has broken my CI enough times its lost any good will it ever had with me.

it mostly breaks CIs or docker builds if you don't hardwire the pip version, because they tend to use latest pip by default..., It's mostly not directly pip's fault, but it can get annoying.

I remember issues when they changed their caching mechanism, or when a couple of libraries I was using were importing internal stuff from pip which got changed in a new version.

Also some packages for some reason need to be installed in the correct order and it's not immediately clear until pip tweaks their installation procedure.

It refuses to install when there are conflicts now.

    $ pip install "six<1.12" "virtualenv==20.0.2" -q
      ERROR: Cannot install six<1.12 and virtualenv==20.0.2       
      because these package versions have conflicting 
      dependencies.

      ERROR: ResolutionImpossible: for help visit   
      https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

People have to write that code ya know?

This replaces a use case of pip freeze. If you only use pip freeze for that use case then it is a replacement.

setup.py (especially if you use declarative setup.cfg) works quite well, you can then use pip-tools to generate requirements.txt which then acts like a lock file.

Frankly I would still be using it if it wasn't that PyPA really trying hard to kill it.

This forced me to try poetry though and is quite decent frankly. I wish it would support building C packages though and I'm missing plugins like setuptools_scm which generates package version from SCM (e.g. git) tags.