undefined | Better HN

story

0 pointscrabbone3y ago0 comments

> once or twice have I ever encountered a showstopper dependency resolution issue.

Hahaha... (rolls on the floor) Do you want to know why that is? No seriously? I'm not laughing at you as much as I'm laughing at Python now, but hey, well, anyways, do you want to know why that happened to you? I know you don't. But I'll tell you anyways!

Until quite recently, pip didn't give a rat's ass if the dependencies it installed were consistent. It would blink a message in the long stream of vomit it spills on the screen saying something like "you have package X installed of version Y, but package Z wants X of version Q, which will not be installed". And happily streamed more garbage to your screen.

It was an issue that was filed against pip for something like 12 years until it got resolved about a year or so ago. Even after it got resolved a lot of people tried to upgrade, saw that that would "break" their deployment, and rolled back to the latest broken version.

Things are sort of improving gradually since then, but we are light years away from the system working properly, and I know you don't want to know why, but I'll tell you anyways!

So, when for whatever reason pip doesn't find a dependency it thinks you need, a lot of packages, when they roll out their "releases", they upload also what Python calls "source release". Which should have never been treated as an installation option, but it is, and is treated like that by default. So, what will happen once pip finally gives up on finding a match, right, you guessed it! -- It's going to try to build it! Installing build dependencies along the way. What you get in the end is anyone's guess, but most likely, it's something broken because the developers who made this release didn't make a release specifically for your version.

Don't despair. There's a flag you can use with pip install that should prevent it from trying to build stuff. But two bad things will happen to you if you use it: in any non-trivial project your dependencies will irreparably break. And, who knows if that flag is implemented correctly... nobody in the real world is using that. So, who knows, maybe it'll format your hard drive along the way.

0 comments

woodruffw3y ago

I understand that you're being hyperbolic for rhetorical purposes, but I think you're overselling the problem with source distributions: most language package ecosystems boil down to the same "baseline" package representation as sdists do, and have the same basic "build it if no binary matches" approach. Many don't even provide built distributions; Rust and Go come to mind.

Python's problem isn't with source distributions as such, but with really bad metadata control (and excessively permissive approaches to metadata generation, like arbitrary Python setup files). Better metadata makes source-based language package management work just fine in every other language's ecosystem; much of the effort in Python packaging over the last ~8 years has been slowly turning Python in that direction.

crabboneOP3y ago

> Python's problem isn't with source distributions as such, but with really bad metadata control

One doesn't preclude the other. I'm not against having a mechanism for automating source installs (like this is done in, eg. RHEL-based distros), but it's insanity if you allow this to happen by default. You may not remember Bumblebee deleting /usr while running some innocuously-looking code during install, but things happen... really bad things...

Things don't need to happen all the time in order for them to be scary. It's enough to have possible catastrophic consequences, even if the event itself is rare.

> Better metadata makes source-based language package management work just fine in every other language's ecosystem

I haven't seen a single one, and I used dozens at this point. This is never a good idea. It's OK to do source installs for development, it's never a good idea to do source installs for deployment. It "works" in other places because of how it's presented (i.e. nobody expects this to be the method of software delivery to the end user). Like, eg. in Cargo (Rust): you, as a developer, download sources and build programs from all the sources on your computer, but your user gets a binary blob they put on the system path and run. It would be insanity and a security nightmare if users were supposed to compile program code before they could run it. The select few who can audit what's being downloaded and how it's been compiled would probably manage, the rest would become victims of all sorts of scams or just random failures propagating beyond their builds into their systems.

> much of the effort in Python packaging over the last ~8 years has been slowly turning Python in that direction.

I'm sorry, but PyPA is managed by clueless people. Whatever they do there only breeds more insanity over time. They neither have a general direction where they want to take the packaging system, nor do they understand the fine details of it. They are also bombarded by insane requirements for useless and harmful features, which they often quick to implement... It's a circus what's going on there. I've lost hope years ago, and now I've become an accelerationist. I just like to see it burn and people run around screaming while their backs are on fire. I get paid to fix this mess. So, PyPAs incompetence is my job security.

muhokutan3y ago

It behaves like a kid you send to the store with a 100 dollars

kapilvt3y ago

I moved to poetry (ergonomics) and publishing wheels with frozen requirements, at least for apps, here's the plugin I used https://github.com/cloud-custodian/poetry-plugin-freeze .. readme has details, tldr freeze the graph for repeatability regardless of tool, Ala pip install works years later.

j / k navigate · click thread line to collapse