undefined | Better HN

0 pointsmplewis5y ago0 comments

You're losing the ability to easily update using fuzzy specs and a lockfile if you're using pip vanilla without pip-tools or poetry.

0 comments

6 comments · 2 top-level

skrtskrt5y ago· 4 in thread

To clarify further - lockfile == reproducible builds without having to pip freeze every single dependency in the tree.

Fuzzy specs == effortless upgrades according to your risk tolerance for a given library (major version for boto3, minor version for pandas, something like that).

Poetry gets you the combination of the two: Let your dep versions float, and easily revert back to a previous deterministic build using the version-controlled lockfile if something breaks.

ghshephard5y ago

I'm wondering why we wouldn't want to pip freeze every single dependency in the tree. I'm looking at our requirements.txt, and everything is of the form:

   vine==1.1.4
   urllib3==1.25.10
   wcwidth==0.1.7

I know that changing the versions of any of the underlying libraries is always a big conversation. (We're still locked in on pandas==0.23.4)

So, if I understand correctly, with Poetry, we might be able to say, "Keep Pandas at 0.23.4 and sqlalchemy at 1.2.7 but figure out all the other dependencies for us and load them."

Or, even better, "Keep Pandas at 0.23.x and sqlalchemy at 1.x.x but figure out all the other dependencies for us and load them."

The advantage here is security patches in underlying libraries come for free, while we focus on porting code for the really high-level + important Libraries which aren't always backwards compatible (Pandas)

Also - if we want to stick with specific versions, that's also possible with the lockfile - so every library will be exactly the same as the one in a build that worked.

The thing I don't understand - is when I do:

   pip install pandas==0.23.4

It does load the dependencies. Indeed, if I create a requirements.txt that just has:

   pandas==1.0.3
   pytz==2020.1
   six==1.14.0

Then pip install -r requirements.txt goes and does:

   $ pip install -r requirements.txt
   Collecting pandas==1.0.3
     Using cached pandas-1.0.3-cp38-cp38-manylinux1_x86_64.whl 
   (10.0 MB)
   Collecting pytz==2020.1
     Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
   Collecting six==1.14.0
     Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
   Collecting python-dateutil>=2.6.1
     Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 
   kB)
   Collecting numpy>=1.13.3
     Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl 
   (14.5 MB)
   Installing collected packages: six, python-dateutil, pytz, 
   numpy, pandas
   Successfully installed numpy-1.19.4 pandas-1.0.3 python- 
  dateutil-2.8.1 pytz-2020.1 six-1.14.0

So - I'm still at a loss of the advantage of poetry vs pip install, given that pip loads dependencies as well - the advantage of "fuzzy specs" seems minimal given it's such a big deal to upgrade the big packages.

lmm5y ago

Nothing locks the version of numpy that you got there. If you run the same thing again in a few weeks you might get a completely different version, and have no way to revert to the version you had before.

xapata5y ago

setup.cfg vs requirements.txt offers this.

skrtskrt5y ago

Poetry is by no means the only option for this.

Lots of people like pip-tools, it would feel a lot more lightweight and closer to pip than Poetry does.

Pipenv exists but... steer clear for a multitude of reasons.

Personally I like that Poetry centers itself around the pyproject.toml standard. I also think that its usability and the enthusiasm of both the maintainers of the users is going to really carry it more into the Python mainstream in the coming years.

2 more replies

ptx5y ago

Pip supports this now. It calls them constraints files: https://pip.pypa.io/en/latest/user_guide/#constraints-files

j / k navigate · click thread line to collapse