undefined | Better HN

0 pointsnickjj5y ago0 comments

How do you deal with keeping your top level dependencies and exact versions of all of your dependencies of dependencies separate in a way that's sane and 100% reproduceable for a typical web app / repo that might not in itself be a Python package?

I'm in the same boat as you in that I'd like to keep using pip but the lack of a lock file is very dangerous because it doesn't guarantee reproduceable builds (even if you use Docker).

In Ruby, Elixir and Node the official package managers have the idea of a lock file. That is the only reason I ever look into maybe switching away from pip.

Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies.

I've been bitten by issues like this so many times in the past with Python where I forgot to define and pin some inner dependency of a tool. Like werkzeug when using Flask. Or a recent issue with Celery 4.3.0 where they forgot to version lock a dependency of their own and suddenly builds that worked one day started to break the next day. These sets of problems go away with a lock file.

0 comments

12 comments · 4 top-level

ghshephard5y ago· 6 in thread

I'm not sure what you mean by, "How do you deal with keeping your top level dependencies and exact versions of all of your dependencies of dependencies separate in a way that's sane and 100% reproduceable for a typical web app / repo that might not in itself be a Python package?"

We use requirements.txt + Docker/k8s to lock in the OS. All of the versions of python modules are defined like:

   six==1.11.0
   sqlalchemy==1.2.7
   squarify==0.3.0

Which locks them to a particular version.

What type of dependencies aren't covered by this (I genuinely am a novice here so would love to be informed where this runs into problems)

nickjjOP5y ago

> What type of dependencies aren't covered by this (I genuinely am a novice here so would love to be informed where this runs into problems)

The dependencies of the dependencies of what you have listed aren't guaranteed to be locked.

For example, let's say for arguments sake you were using celery.

When you install celery, by default these deps will also be installed: https://github.com/celery/celery/blob/master/requirements/de...

Those aren't very locked down. If you install celery today you might get vine 5.0.0 but in X months from now you might get 5.9.4 which could have backwards compatibility issues with what celery expects.

So now you build your app today and everything works but X months from now you build your same app with the same celery version and things break because celery isn't compatible with that version of vine.

This happened a few months ago. Celery 4.3.0 didn't version lock vine at all and suddenly all celery 4.3.0 versions broke when they worked in the past. That was tracked at https://github.com/celery/celery/issues/3547.

Docker doesn't help you here either because your workflow might be something like this:

- Dev works locally and everything builds nicely when you docker-compose build

- Dev pushes to CI

- CI builds new image based on your requirements.txt

- CI runs tests and probably passes

- PR gets merged into master

- CI kicks in again and builds + tests + pushes the built image to a Docker registry if all is good

- Your apps use this built image

But there's no guarantee what you built in dev ends up in prod. Newer versions of certain deps could have been built in CI. Especially if a PR has been lingering for days before it gets merged.

A lock file prevents this because if a lock file is present the lock file gets used, so if you built and included a lock file in version control, then CI will build what you pushed from dev, so the chain is complete from dev to prod for guaranteeing the versions you want. That is how it works with Ruby, Elixir, Node and other languages too. They have 2 files (a regular file where you put your top level deps and a machine generated lock file). A lock file in Python's world would translate to what pip3 freeze returns.

ghshephard5y ago

Thanks very much - your description of our workflow is really good (and is pretty close to exactly what we have!)

I don't understand what dependencies everyone keeps talking about (which seems to be a big deal with Poetry) - when you run: pip freeze

It captures every single python module, dependencies as well. Because everything in the dependencies file is listed as: aaaaaaa==xy.z

You are guaranteed to have the exact same version.

We have all sorts of turf wars when someone wants to roll forward the version of a module, and, in the case of the big ones (Pandas) we sometimes hold off for 6-9 months before rolling it forward.

But there is something that Poetry is doing that is better than "pip freeze" - I think once I figure that out, I'll have an "aha" moment and start evangelizing it. I just haven't got there yet.

JerryJeffJeff5y ago

You need a constraints file

https://pip.pypa.io/en/stable/user_guide/#constraints-files

pip install celery==4.3.0 -c constraints.txt

Where constraints.txt defines exact versions for everything

turingcomplet5y ago

Why not keep the same docker image throughout then lifecycle? E.g. merge to dev branch, trigger ci (build image at this point), maybe deploy to a test environment, run more tests, then deploy to prod. No chance of packages changing since the image isn't rebuilt. Of course if not using docker, a lock file (i.e. actual dependency resolution) would seem essential for reproducibility.

jshen5y ago

First, how did you generate that requirements file?

Second, how do you separate dev dependencies from prod dependencies, and how do you update a dependency and ensure all of its transitive dependencies are resolved appropriately?

ghshephard5y ago

    pip freeze > requirements.txt

Lists every python module that's been loaded into the virtualenvironment. So, from my (admittedly new) understanding, that means we guarantee that in the production/devel/docker environment - every python module will be identical to whatever was installed in the virtual env.

Dependencies and transitive dependencies are guaranteed to be resolved/ensured because we list everyone one of them out in the requirements.txt file.

1 more reply

ptx5y ago· 2 in thread

There is one! Pip supports, since a few years back, a constraints file (i.e. a lock file) that does just this.

This is a nice guide on how to use it: https://www.lutro.me/posts/better-pip-dependency-management

dalemyers5y ago

This isn't quite the same. Yes, I can update my root dependencies in the `requirements.txt` file, run `pip install -r requirements.txt` and the run `pip freeze > requirements.txt`, but that's convoluted and requires me to know exactly what my root dependencies are. Is `astroid` something our tools use directly, or is it just a dependency of `pylint`? It's not clear. A lockfile clears this up.

ptx5y ago

Yes, and in addition to the requirements file, pip supports a constraints file, which is the lockfile you describe. It's separate from the requirements file. It solves exactly this problem.

Docs: https://pip.pypa.io/en/stable/user_guide/#constraints-files

3 more replies

beaugunderson5y ago

> How do you deal with keeping your top level dependencies and exact versions of all of your dependencies of dependencies separate in a way that's sane and 100% reproduceable for a typical web app / repo that might not in itself be a Python package?

`pip-compile` from `pip-tools` is my go-to for this.

xapata5y ago

> Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies.

Use setup.cfg to define your top level dependencies. Use requirements.txt as your "lock" file. But even then you won't get reproducible builds across different OSes, or with different non-Python things installed on your machines. Use Docker images to guarantee staging and production will be identical.

j / k navigate · click thread line to collapse

0 comments

12 comments · 4 top-level

ghshephard5y ago· 6 in thread

We use requirements.txt + Docker/k8s to lock in the OS. All of the versions of python modules are defined like:

   six==1.11.0
   sqlalchemy==1.2.7
   squarify==0.3.0

Which locks them to a particular version.

What type of dependencies aren't covered by this (I genuinely am a novice here so would love to be informed where this runs into problems)

nickjjOP5y ago

> What type of dependencies aren't covered by this (I genuinely am a novice here so would love to be informed where this runs into problems)

The dependencies of the dependencies of what you have listed aren't guaranteed to be locked.

For example, let's say for arguments sake you were using celery.

When you install celery, by default these deps will also be installed: https://github.com/celery/celery/blob/master/requirements/de...

Docker doesn't help you here either because your workflow might be something like this:

- Dev works locally and everything builds nicely when you docker-compose build

- Dev pushes to CI

- CI builds new image based on your requirements.txt

- CI runs tests and probably passes

- PR gets merged into master

- CI kicks in again and builds + tests + pushes the built image to a Docker registry if all is good

- Your apps use this built image

But there's no guarantee what you built in dev ends up in prod. Newer versions of certain deps could have been built in CI. Especially if a PR has been lingering for days before it gets merged.

ghshephard5y ago

Thanks very much - your description of our workflow is really good (and is pretty close to exactly what we have!)

I don't understand what dependencies everyone keeps talking about (which seems to be a big deal with Poetry) - when you run: pip freeze

It captures every single python module, dependencies as well. Because everything in the dependencies file is listed as: aaaaaaa==xy.z

You are guaranteed to have the exact same version.

We have all sorts of turf wars when someone wants to roll forward the version of a module, and, in the case of the big ones (Pandas) we sometimes hold off for 6-9 months before rolling it forward.

But there is something that Poetry is doing that is better than "pip freeze" - I think once I figure that out, I'll have an "aha" moment and start evangelizing it. I just haven't got there yet.

JerryJeffJeff5y ago

You need a constraints file

https://pip.pypa.io/en/stable/user_guide/#constraints-files

pip install celery==4.3.0 -c constraints.txt

Where constraints.txt defines exact versions for everything

turingcomplet5y ago

jshen5y ago

First, how did you generate that requirements file?

Second, how do you separate dev dependencies from prod dependencies, and how do you update a dependency and ensure all of its transitive dependencies are resolved appropriately?

ghshephard5y ago

    pip freeze > requirements.txt

Dependencies and transitive dependencies are guaranteed to be resolved/ensured because we list everyone one of them out in the requirements.txt file.

1 more reply

ptx5y ago· 2 in thread

There is one! Pip supports, since a few years back, a constraints file (i.e. a lock file) that does just this.

This is a nice guide on how to use it: https://www.lutro.me/posts/better-pip-dependency-management

dalemyers5y ago

ptx5y ago

Yes, and in addition to the requirements file, pip supports a constraints file, which is the lockfile you describe. It's separate from the requirements file. It solves exactly this problem.

Docs: https://pip.pypa.io/en/stable/user_guide/#constraints-files

3 more replies

beaugunderson5y ago

`pip-compile` from `pip-tools` is my go-to for this.

xapata5y ago

> Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies.

j / k navigate · click thread line to collapse