undefined | Better HN

0 pointsaldanor3y ago0 comments

- Managing non-python / binary dependencies alongside python ones

- Managing a mixture of python / binary dependencies in local registries

- conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally

0 comments

9 comments · 2 top-level

nl3y ago· 5 in thread

These are all legitimate reasons, but my personal experience (and perhaps preference?) is to use Docker for anything that is more complex than pip can handle.

> conda-forge managing builds of some really flaky binary python packages that are sometimes a nightmare to build locally

Yeah this is fair. Fortunately it's becoming rarer.

aldanorOP3y ago

Well. To each their own, use cases differ so wildly it's hard to compare them.

The key audience for conda is ML/DS space, where most if not all packages come from either C/C++/Rust/Fortran and have to be compiled, while also requiring a consistent set of external C libraries like libblas, etc. As I said, some of those packages are a completely nightmare to build locally. Conda simplifies this by a lot in that you can just 'conda create -n myenv some=1.0 crazy=2.0 deps=2.0' and in a few seconds (if you use mamba and not conda) you have a working Python environment so off you go; no dockers, no local builds etc.

techwizrd3y ago

Honestly, I've found that conda has made operationalizing code very difficult. We've found it much easier to simply switch back to using pip, poetry, docker, and the standard OS package management tools rather than conda. Conda's dependency resolution is also quite slow and causes our builds & CI to timeout unless we drop in mamba.

1 more reply

thrdbndndn3y ago

Can these hard-to-build-locally lib just upload a wheel to PYPI?

hobs3y ago

lol, the dependency hell of even using mamba can be large on old things, I have waited more than an hour for dependency resolution using both.

PaulHoule3y ago

Seems docker is going from the frying pan to the fire. Have they added ‘resume download’ yet to docker? Over my slow DSL I can’t stand how docker makes me download 7G of images when I want to install something very simple, frequently it fails to do the download and I have to do it several times so it adds up to more like 28G of downloading and all that waiting.

I worked at one place where management was shocked when I told them the image build process would take 20 minutes on gigabit fiber up in Canada and we agreed to time it and I measured 18 minutes. Docker slows down “dev” to the speed of “ops.”

I don’t know how they did it but the data scientists could always find f-ed up Python images, you never got the same default character encoding twice, one time the default character set was Hungarian and I wonder how that happens…

drdaeman3y ago· 2 in thread

Doesn't pip (with wheels) solve this all just fine? Or I'm missing something?

silveraxe933y ago

You're missing something. - Managing non-python / binary dependencies alongside python ones

pip with wheels doesn't deal with non-python packages. I used to be in a horrible locked down corpo laptop. Conda was invaluable in getting stuff to run, like chromedriver, etc.

drdaeman3y ago

Uh, I guess I do? I mean, there is https://pypi.org/project/chromedriver-binary/ (and a lot of similar packages)...

I honestly don't remember which one I've used for chromedriver when I needed it for my project, but I've surely installed all the stuff with "just" pip/poetry. Larger projects are typically packaged like this, with setup.py performing the downloads, while wheels solve the problem with Python libraries with native dependencies (e.g. how psycopg-binary works).

Maybe Conda makes it slightly more convenient, but I've always treated pip as the standard Python package management tool (it's a part of the standard library now, after all) and Conda was always "that weird non-standard thing some folks use for some odd reason" for me.

j / k navigate · click thread line to collapse