Conda: A package management disaster? (opens in new tab)

In this sense I personally prefer pixi because of this. It is pixi like but resolves using conda channels like conda, and similar to conda it supports PyPI packages via uv.

With a background in scientific computing where many of the dependencies I managed are compiled, conda packages gives me much more control.

P.S. I’d like to point out to others to differentiate between package index and package managers. PyPI is an index (that hosts packages in a predefined format) while pip, poetry, uv are package managers that resolve and build your environments using the index.

Similarly but a bit more confusingly, conda can be understood as the index, hosted by anaconda but can also be hosted elsewhere, with different “channels” (kinda like a GitHub organization) where conda-forge is a popular one built by communities. Conda is also a reference implementation of a package manager that uses anaconda channels to resolve. Mamba is an independent, performant, drop in replacement of conda. And pixi is a different one with a different interface by the author of mamba.

Even more confusingly, there are distributions. Distributions come with a set of predefined packages together with the package manager such that you just start running things immediately (sort of like a TeXLive distribution in relation to the package manager tlmgr.) there are anaconda distributions (if you installed anaconda instead of installing conda, that’s what you get), but also Intels distribution for Python, mini forge, mambaforge, etc.

Is there a reason this behaviour couldn’t be implemented in uv?

Is this beyond what the pyproject.toml spec supports?

This was mostly because most scientific packages didn't provide Windows binary builds for many years.

Today you can just "pip install scipy" on Windows at it will just work.

It also made a bunch of neural net libraries much easier to install before containers became popular.

My experience with conda is that its fine if you're the original author of whatever you're using it for and never share it with anyone else. But as a professional I usually have to pull in someone else's work and make it function on a completely different machine/environment. I've only had negative experiences with conda for that reason. IME the hard job of package management is not getting software to work in one location, but allowing that software to be moved somewhere else and used in the same way. Poetry solves that problem, conda doesn't.

Poetry isn't perfect, but it's working in an imperfect universe and at least gets the basics (lockfiles) correct to where packages can be semi-reproducible.

There's another rant to be had at the very existence of venvs as part of the solution, but that's neither poetry or anaconda's fault.

You are doing something right, author does some pretty unusual things:

- Setup custom kernels in Jupyter Notebook

- Hardlink the environments, then install same packages via pip in one and conda in others

- install conda inside conda (!!!) and enter nested environment

- Use tox within conda

I believe as long as you treat the environments as "cattle" (if it goes bad, remove it and re-create from yaml file), you should not have any problems. It's clearly not the case of for the post's author though.

As someone with a formal computer science, half of my friends who work in other sciences have asked me to help them fix their broken conda environments

This is exactly the kind of thing that causes python package nightmares. Pip is barely aware of packages it's installed itself, let alone packages from other package managers and especially other package repositories. Mixing conda and pip is 100% doing it wrong (not that there's an easy way to do it right, but stick to one or the other, I would generally recommend just using pip, the reasons for conda's existance are mostly irrelevant now)

I had the same experience. But you should try pixi, which is to conda what uv is to pip.

God forbid you should require conda-forge and more than three packages lest the dependency resolver take longer than the heat death of the planet to complete.

Same but I try to use conda to install everything first, and only use pip as a last resort. If pip only installs the package and no dependency it's fine

i think you got lucky and fell into best practices on your first go

> except for any binaries or CUDA related things from conda

doing the default thing with cuda related python packages used to often result in "fuck it, reinstall linux". admittedly, i dont know how it is now. i have one machine that runs python with a gpu and it runs only one python program.

1. You need to run export manual while other tools you mentioned would create it automatically (the lock file) 2. Distinguishes between direct dependencies (packages you added yourself) and indirect dependencies (packages of the packages)

In your requirements.txt, do you pin the concrete versions or leave some leeway?

If you aren't precise, you're gonna get different versions of your dependencies on different machines. Oops.

Pinning concrete versions is of course better, but then there isn't a clear and easy way to upgrade all dependencies and check whether ci still passes.

> For personal projects venv + requirements.txt has never caused problems for me.

That means you don't use Windows.

What is great. Keep not using it. But most people will have a different experience.

For using packages, venv + requirements.txt works, but is a bit clunky and confusing. Virtual environments are very easy to break by moving them or by updating your OS (and getting a new Python with it). Poetry is one alternative, but there are far too many options and choices to make. For building packages, there are similarly many competing options with different qualities and issues.

i think there might be merit to gdiamos's point that python is a popular language with a large number of users, and this might mean that python package management isn't unusually bad, but more users implies more complaints.

i think there was a significant step change improvement in python packaging around 2012, when the wheel format was introduced, which standardised distributing prebuilt platform-specific binary packages. for packages with gnarly native library dependencies / build toolchains (e.g. typical C/fortran numeric or scientific library wrapped in a layer of python bindings), once someone sets up a build server to bake wheels for target platforms, it becomes very easy to pip install them without dragging in that project's native build-from-source toolchain.

venv + pip (+ perhaps maintaining a stack of pre-built wheels for your target platform, for a commercial project where you want to be able to reproduce builds) gets most of the job done, and those ingredients have been in place for over 10 years.

around the time wheel was introduced, i was working at a company that shipped desktop software to windows machines, we used python for some of the application components. between venv + pip + wheels, it was OK.

where there were rough edges were things like: we have a dep on python wrapper library pywhatever, which requires a native library libwhatever.dll built from the c++ whatever project to be installed -- but libwhatever.dll has nothing to do with python, maybe its maintainers kindly provide an msi installer, so if you install it into a machine, it gets installed into the windows system folder, so venv isn't able to manage it & offer isolation if you need to install multiple versions for different projects / product lines, as venv only manages python packages, not arbitrary library dependencies from other ecosystems

but it's a bit much blame python for such difficulties: if you have a python library that has a native dependency on something that isnt a python package, you need to do something else to manage that dep. that's life. if you're trying to do it on windows, which doesn't have an O/S level package manager.. well, that's life.

Try building a package and you will get hundreds of little paper cuts. Need a different index for some packages? It will work with a cli "pip install -from-index", but pip will not let you add an index in a requirement.txt for... security reasons. That means, good luck trying to "enforce" the CUDA version of pytorch without using third party tooling. So you either hard code a direct link (breaks platform portability), as that will work, or give up trying to make your project installable with "Pip install " Or "python build". Remember, pytorch basically has no CUDA builds anymore in its pypi index and no way to get CUDA torch from there (but I think this might have changed recently? )

Oh, and if some package you are using has a bug or something that requires you to vendor it in your repo, well then good luck because again, PEP 508 does not support installing another package from a relative link. You either need to put all the code inside the same package, vendored dependency included, and do some weird stuff to make sure that the module you wanted to vendor is used first, or... you just have to use the broken package, again for some sort of security reasons apparently.

Again, all of that might even work when using pip from the cli, but good luck trying to make a requirements.txt or define dependencies in a standard way that is even slightly outside of a certain workflow.

venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this.

Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding the benefits.

I can sort of the the argument, if you really really need to lock down your dependencies to very specific version, which I don't recommend you do.

For development I use venv and pip, sometimes pyenv if I need a specific Python version. For production, I install Python packages with apt. The operating system can deal with upgrading minor library versions.

I really hate most other package managers, they are all to confusing and to hard to use. You need to remember to pull in library update, rebuild and release. Poetry sucks too, it's way to complicated to use.

The technical arguments against Python packages managers are completely valid, but when people bring up Maven, NPM or even Go as role models I check out. The ergonomics of those tools are worse than venv and pip. I also think that's why we put up with pip and venv, they are so much easier to use than the alternative (maybe excluding uv). If a project uses Poetry, I just know that I'm going to be spending half a day upgrading dependencies, because someone locked them down a year ago and there's now 15 security holes that needs to be plugged.

No, what Python needs is to pull in requests and a web framework into the standard library and then we can start build 50% of our projects without any dependencies at all. They could pull in Django, it only has two or three dependencies anyway.

It's popular enough that it causes pain for a lot of people.

Coming from C++, IMO, it is vastly better.

What do you do though if you want to import code written in C++? Especially complex, dependency-heavy like CUDA/ML stuff?

You can just give up and say that "The proper way to do this is to use the Nvidia CUDA toolkit to write your cuda app in C++ and then invoke it as a separate process from node" [0]. That apparently works for node, but Python wants much more.

If you actually want to use high-performance native code in your slow compiled language, then no solution is going to be very good, that's because the problem is inherently hard.

You can rely on host OS as much as possible - if OS is known, provide binaries; if it's unknown, provide source code and hope user has C/C++/Rust/Fortran compilers to build it. That's what uv, pip, etc.. do.

You can create your own parallel OS, bringing your own copy of every math libray, as well as CUDA even if there are perfectly good versions installed on the system - that's what conda/minconda does.

You can implement as much as possible in your own language, so there is much less need to use "high-performance native language" - that's what Rust and Go do. Sadly, that's not an option for Python.

[0] https://stackoverflow.com/questions/20875456/how-can-i-use-c...

This is spot on. Running some python projects on nixos is a nightmare because of this model. Especially if it’s ML related.

In Python you need to deliberately mess with the system Python by running your package installer under sudo. That’s not something you do accidentally.

When dealing with the system Python you should always use the system package manager. That extends to Macs for both brew and MacPorts.

Looks like the webpage is on GitHub too: https://github.com/pyherald/pyherald/blob/main/docs/articles...

Back online

If we're doing anecdotal evidence, then mine is that conda is by far the worst of the main Python packaging solutions in use. The absurd slowness and incompatibility with the entire rest of the Python world are only the visible tip of that iceberg. To the best of my ability to tell, conda largely exists to make up for endemic deficiencies in Windows software distribution toolchains (not Python specific) and sadly it's not even good at that either.

Mind you, glad it works for you. Warms my grey heart to know there's some balance in this universe. :)

It's because of the SAT solver for dependencies. Unlike Pip, it keeps track of every package you installed and goes out of its way to avoid installing incompatible packages.

Why go through all this trouble? Because originally it was meant to be a basic "scientific Python" distribution, and needed to be strict around what's installed for reproducibility reasons.

It's IMO overkill for most users, and I suspect most scientific users don't care either - most of the time I see grads and researchers just say "fuck it" and use Pip whenever Conda refuses to get done in a timely fashion.

And the ones who do care about reproducibility are using R anyway, since there's a perception those libraries are "more correct" (read: more faithful to the original publication) than Pythonland. And TBH I can't blame them when the poster child of it is Sklearn's RandomForestRegressor not even being correctly named - it's bagged trees under the default settings, and you don't get any indication of this unless you look at that specific kwarg in the docs.

Personally, I use Conda not for reproducibility, but so all of my projects have independent environments without having to mess with containers

The situation has since changed as the solver is rewritten. (Upstreamed from the work done in mamba.) I encourage you to try again.

Try it again. It now uses libmamba for solving dependencies

Do you have a publicly available copy of your flake?

pip + venv or just using Poetry usually is 100% headache-free for me. Conda, however, is usually a great way to ensure I have an awful time.

dnf install python3-gdal?

I wish you luck with tracking down versions of software used when you're writing papers... especially if you're using multiple conda environments. This is pretty much the example used in the article -- version mismatches.

But, I think this illustrates the problem very well.

Conda isn't just used for Python. It's used for general tools and libraries that Python scripts depend on. They could be C/C++ that needs to be compiled. It could be a Cython library. It could be...

When you're trying to be a package manager that operates on-top of the operating system's package manager, you're always going to have issues. And that is why Conda is such a mess, it's trying to do too much. Installation issues are one of the reason why I stopped writing so many projects in Python. For now, I'm only doing smaller scripts in Python. Anything larger than a module gets written in something else.

People here have mentioned Rust as an example of a language with a solid dependency toolchain. I've used more Go, which similarly has had dependency management tooling from the begining. By and large, these languages aren't trying to bring in C libraries that need to be compiled and linked into Python accessible code (it's probably possible, but not the main use-case).

For Python code though, when I do need to import a package, I always start with a fresh venv virtual environment, install whatever libraries are needed in that venv, and then always run the python from that absolute path (ex: `venv/bin/python3 script.py`). This has solved 99% of my dependency issues. If you can separate yourself from the system python as much as possible, you're 90% of the way there.

Side rant: Which, is why I think there is a problem with Python to begin with -- *nix OSes all include a system level Python install. Dependencies only become a problem when you're installing libraries in a global path. If you can have separate dependency trees for individual projects, you're largely safe. It's not very storage efficient, but that's a different issue.

And then somebody tries to install mamba via conda and that house of cards reveals itself.

It kind of limits its own blast radius that way.

> what makes this so difficult to solve in Python?

Python creates the perfect storm for package management hell:

- Most the valuable libraries are natively compiled (so you get all the fun of distributing binaries for every platform without any of the traditional benefits of native compilation)

- The dynamic nature makes it challenging to understand the non-local impacts of changes without a full integration test suite (library developers break each other all the time without realizing it, semantic versioning is a farce)

- Too many fractured packaging solutions, not a single one well designed. And they all conflict.

- A bifurcated culture of interactive use vs production code - while they both ostensibly use the same language, they have wildly different sub-cultures and best practices.

- Churn: a culture that largely disavows strong backwards compatibility guarantees, in favor of the "move fast and break things" approach. (Consequence: you have to move fast too just to keep up with all the breakage)

- A culture that values ease of use above simplicity of implementation. Python developers would rather save 1 line of code in the moment, even if it pushes the complexity off to another part of the system. The quite obvious consequence is an ever-growing backlog of complexity.

Some of the issues are technical. But I'd argue that the final bullet is why all of the above problems are getting worse, not better.

The problem is that Python refuses to take responsibility for the whole ecosystem. One of the biggest success stories in programming language development has been Rust's realization that all of it matters: language, version management, package management, and build tools. To have a truly outstanding experience you need to take responsibility for the whole ecosystem. Python and many other older languages just focus on one part of the ecosystem, while letting others take care of different parts.

If Python leadership had true visionaries they would sit down, analyze every publicly available Python project and build a single set of tools that could gradually and seamlessly replace the existing clusterfuck.

Python developers will pretend the language is all about simplicity and then hand you over to the most deranged ecosystem imaginable. It sure is easy to pretend that you have a really simple ecosystem when you cover your eyes and focus on a small segment of the overall experience.

> What I don't understand - what makes this so difficult to solve in Python?

I think there are many answers to this, and there are many factors contributing to it, but if I had to pick one: The setup.py file. It needs to be executed to determine the dependencies of a project. Since it's a script, that allows any maintainer of any package you are using to do arbitrarily complex/dumb stuff in it like e.g. conditionally adding dependencies based on host system specific environment markers, or introduce assumptions on the environment it is being installed to. That makes trying to achieve all the things you'f want from a modern package manager so much harder.

This also means that the problem isn't just concentrated in 1-2 central package management projects, but scattered throughout the ecosystem (and some of the worst offenders are some of Python's most popular sub-ecosystems).

There is some light with the introduction of the pyproject.toml, and now uv as a tool taking advantage of it.

I think at least part of it is that there are so many solutions for Python packaging, which are often intermixed or only half-supported by developers. It's a tough ask to provide dedicated support for pip, conda, poetry and what else is there plus a couple different ways to create virtual environments. Of course if you do everything right, you set it up once (if even that) and it just keeps working forever, but it is never like that. Someone will use a tool you haven't and it will not work correctly and they will find a workaround and the mess starts.

Also I think that Python packages are sometimes distributed as shared libraries is a problem. When I think about conan or vcpkg (package managers for C and C++), they usually suck because some dependencies are available on some platforms and not on others or even in one version on one platform and in another version on another and you get messes all around if you need to support multiple platforms.

I think generally binary package managers are almost always bad* and source based package managers almost always work well (I think those are essentially easy mode).

*: unless they maintain a source package of their own that they actually support and have a fixed set of well-supported platforms (like system package managers on most Linux distros do).

A lot of path dependency, but essentially

  1. A good python solution needs to support native extensions. Few other languages solve this well, especially across unix + windows.
  2. Python itself does not have package manager included.

I am not sure solving 2 alone is enough, because it will be hard to fix 1 then. And ofc 2 would needs to have solution for older python versions.

My guess is that we're stuck in a local maximum for a while, with uv looking like a decent contender.

Nix solves it for me. Takes a bit more effort upfront, but the payoff is "Python dependency determinism," which is pretty much unachievable in any other way, so...

Agreed. But the problem is now fully solved by https://docs.astral.sh/uv/.

Charlie Marsh (one of the people behind astral.sh, the creators of ruff and uv) explains it well in this talk:

https://www.youtube.com/watch?v=zOY9mc-zRxk

It is not a new discovery that Python is terrible for packaging and distribution. Unfortunately, very little has been done about this. The fact that Python is used on particular environments controlled by the developers, mainly machine learning, makes this even more difficult to fix.

> what makes this so difficult to solve in Python?

I think the answer is the same thing that makes it difficult to make a good package manager for C++.

When a language doesn't start with decent package management, it becomes really hard to retrofit a good one later in the lifespan of that language. Everyone can see "this sucks" but there's simply no good route to change the status quo.

I think Java is the one language I've seen that has successfully done the switch.

Python packaging’s complexities are difficult to attribute to any single cause. But path dependency, extremely broad adoption, and social conventions with the Python community (which has historically preferred standards over picking single tools) are all contributing factors.

Most of these aspects have significantly improved over the last decade, at least for the standard packaging ecosystem. I don’t know about Conda, which has always been its own separate thing.

PyPI was always broken due to weird ideas for problems that were long solved in other languages or distributions. They had/have the backing of fastly.net, which created an arrogant and incompetent environment where people listed to no one.

Conda suffers from the virtual environment syndrome. Virtual environments are always imperfect and confusing. System libraries sometime leak through. The "scientific" Python stack has horrible mixtures of C/C++/Cython etc., all poorly written ad difficult to build.

Projects deteriorated in their ability to build from source due to the availability of binary wheels and the explosion of build systems. In 2010 there was a good chance that building a C project worked. Now you fight with meson versions, meson-python, cython versions, libc versions and so forth.

There is no longer any culture of correctness and code cleanliness in the Python ecosystem. A lot of good developers have left. Some current developers work for the companies who sell solutions for the chaos in the ecosystem.

Python packaging is broken mostly because bootstrapping is broken, and it cascades to packaging but people don't know the bootstrapping is responsible and blame packaging.

Not saying packaging doesn't have faults, but on it's own, on a good Python setup, it's actually better than average. But few people have a good setup. In fact most people don't know what a good setup looks like.

And here is why bootstrapping is broken: https://www.bitecode.dev/p/why-is-the-python-installation-pr...

>> One thing which I think messed this up from the beginning was applying the Unix philosophy with several/many individual tools as opposed to one cohesive system

Well, Unix IS the cohesive system..

my approach is to ignore all the *conda stuff and:

yay -S python-virtualenv # I'm on arch, do not confuse with 12 similarly named alternatives pyenv virtualenv 3.10 random-python-crap pyenv local 3.10.6/envs/random-python-crap pip install -r requirements.txt

and it works (sometimes deps are in some other places, or you have to pass -c constraints.txt or there is no file and you need to create it in various ways)

At least by not using local .env directories, I always know where to find them.

I install a lot of AI projecst so I have around 1TB just for the same python dependencies installed over and over again.

Sometimes I can get away with trying to use the same venv for two different projects but 8/10 deps get broken.

python developers run pyenv inside of docker containers.. they just have no clue what good dependency management could even possibly look like

I don’t think the Python community has a culture of thinking about software engineering in a principled and systematic way like you would see in places like Haskell, Rust or Clojure communities.

Pythons strength (and weakness) is an emphasis on quick scripts, data science and statistics.

There’s simply not the right people with the right mindset.

or lua rocks

Seconded. How about we don't write a blog trashing an implementation of something when our own design is missing some very basic accessibility and ux features.

Though I agree with the premise, Conda is an absolute pest when you start customising an environment with a number of packages. Dependency resolution hell.

How does one even do that? Browsers try quite hard to not break words and make text readable by default.

I see this on Firefox on desktop too. I usually try not to criticize the presentation of things posted here, but this is completely unreadable. I've tried several times to get through the first couple paragraphs, but it's just not worth the extra mental effort.

Even on Desktop.

When 30% of your page is junk, it makes you wonder about the other 30%...

You might the linked src.

https://mail.python.org/pipermail/python-list/2024-May/91230...

[oof, never mind, it's worse in some ways]

I gave up after a few lines. Why would you do this?

fixed!