It’s so thoroughly broken, every day a dev on some team gets their poetry env entangled with some system installed python, or numpy suddenly decides all the CI builds will now compile it from scratch on every build, or.. Today it was segfaults on poetry version X on the M1 Mac’s, that went away in version Y but of course version Y broke pandas for the windows devs..
Save in requirements.txt. Includes sub dependencies that are system specific.
Install from requirements.txt on different machine and get errors.
Uninstall dependency and save to requirements.txt.
Look at requirements.txt and see that sub decencies are still there.
What is the right way to avoid these issues on Python?
Once you set the version of python in the virtualenv and activate it, the right python will always be there. The virtualenv even has an instance of python in it's bin. When that python is run, whatever's installed in the virtualenv comes with it.
Poetry, however, does suck even though it's using virtualenv underneath. It's doing frankly too much and seems to have a lot of regressions. In the end it should only be looking at the contents of the virtualenv and using pip to install the correct packages into it.
Edited to add: With the last version of poetry causing issues in my development env, I'm about ready to ban it from our team.
So to fix it you would need to change that, but then you would break a load of Python libraries which are most of the reason people use Python in the first place.
Deno doesn't really have that issue so badly because JS imports are ... not completely sane but they're quite a lot saner.
Anyway I think if you were going to break compatibility with Python by fixing the packaging and import system, it's not a much bigger step to fix the rest of Python too and then you have Nim or Lobster.
I built this resistance to docker about 5 years ago, is it (or was it) still correct? Is there a best practice for sharing GPUs with containers that's not a pain in the ass?
Edit: I should specify that I typically develop on Ubuntu, but occasionally will do so on Windows (without WSL).
Python doesn't need "something like deno". All programming languages need package management.
> And it works well on Windows. No idea about Mac and don't care.
Package management is a cross platform problem. You may not need it but that doesn’t make the current solution good.
This is where I kinda regret switching from and miss chef. Pick up an RPM/APT package from vendor, shove it into your package manager, the environment works. It developed a sufficient amount of weird behaviors later on, but that's besides the point.
Instead we have a really precise documentation of 4-5 steps to get everything installed in a way that should overall work, and each step is annotated with 4-5 ways that look deceptively good but end up being even more quirky. And when you do all of that, and review it twice, it still doesn't even work consistently across 6 workstations. 3 work the same, 2 have interpreter discovery anomalies, 1 has import conflicts with seemingly unrelated other installed stuff.
Whenever I have a working python VM / container build or installation, I kinda feel a need to shut up about my jenga tower because otherwise the universe will find a way specifically to ruin my day.
I hate how I have to feel like this about a language I really like.
And then
Python3 -m venv envdir
source envdir/bin/activate
bin/python3 -m pip install --upgrade pip
And similarly the rest of the packages needed for the project
This does not mess up the system library or other python projects
This step is the least intuitive until you've done it enough times that it's muscle memory.
Can it be better--yes. Is it worse than _any_ other language's packaging if it were to scale to the number/kind of cases Python supports/used--no.
"thoroughly broken" is useless. Are there bugs--most definitely yes. Do report them, or even help fixing them if you care.
There are many many different use-cases where different approaches are preferable (no size fits all).
Personally, most most of the time packaging is a non-issue e.g., pinning versions and cache are highly recommended for CI to avoid unnecessary surprises, flakiness.
- the guy who created C++
I always bring up Go in such arguments - according to the survey [0] done by the Go team, 93% of Go users are satisfied with the language.
In Python land one of those seems to come along every couple of years. They start over and implement a solution for some subset of the problem space. Then they realize the Python packaging mess is much, much bigger than anticipated and progress stalls. At that point there are 15 partial "standard" packaging solutions for Python, where there were 14 before. None of them are feature complete, and they don't really coexist well or at all.
Cue the obligatory https://xkcd.com/927/ and https://xkcd.com/1987/
From that presepective , most python libs are written in python only.
If you get any python job, you'd get a lot of side-eyes for not knowing of/how to use and initialize basic venv package sets. Even if the shop uses conda.
People were not impressed.
Hence the previous article (https://bitecode.substack.com/p/relieving-your-python-packag...) advocating to go back to the basics yourself.
See the reddit thread to witness exactly this in action:
https://www.reddit.com/r/Python/comments/124hktv/comment/je5...
Now we have a never ending train of hotshot developers that think they can "solve" Python packaging with another tool, without realizing that you can't design a tool to fix a broken 15-year-old system. It's not just a "there are now X competing standards" issue, it's also the fact that anyone who makes an earnest attempt is doomed. They will never be able to get it past the point of universally good enough.
I lament the fact that I'm dragged back into the Python ecosystem every time I need to do something with a PyTorch-based tool someone spent years of effort building. I would never recommend the language to a newcomer. Ruby pollutes the global namespace, and Gemfiles execute arbitrary code, but at least with Ruby there's Gem, and then there's Bundler, and nobody complains.
Will we forever be beholden to the mistaken decisions of the original developers in the early 90's, just because Python is simply too widespread? Because even people like me who detest the packaging experience are forced to go back there because nobody's going to rewrite the hottest ML project in Elixir or something? Even the simple mistakes like putting venv files in Scripts/ instead of bin/ forcing one to work around it in cross-platform CI make no sense in hindsight. Nobody remembers why it was made that way after so long. Now it just floats over you like a spectre.
Python's environment management is just the tip of the ice berg
Dynamic typing
Gil
Speed is slow at best
Async streams having its set of non tractable issues
Debugging by printing pdb all over and breaking
I would recommend to 'python -m venv' and thats all.
The only limitation I've encountered is when moving the environment or renaming one of the parent directories. In which case it's easy to create a new one:
# optional: freeze the environment if you don't already have a requirements.txt
source .venv/bin/activate
pip freeze > requirements.txt
deactivate
# remove the old environment
rm -rf .venv
# create a new one
python3.10 -m venv .venv
# activate it
source .venv/bin/activate
# install the requirements
pip install -r requirements.txt # install the requirements
npm install
# remove it
rm -rf node_modules
Aside from needing to know to use it. Which is certainly a problem. But python blessing a single venv-system might be worse in the long run...?
The whole reason I posted this one is to justify the "just use -m pip and -m venv" stance I wrote in the article 4 days ago.
Because you will always have a very vocal minority of people that will fill the threads with counter arguments actually increasing complexity and risks of failure.
As the article says, I think these days it's pretty safe to just use Python's built-in venv and stay away from everything else.
sudo apt install python3.X
sudo apt install python3.X-venv
python3.X -m venv .venvNothing in computing is "simply".
"Wellllll, that's non-trivial."
It’s not confusing if you short-circuit the conversation; if you need advice, don’t go down the rabbit hole, just use poetry and pyenv on MacOS, you will make your life easier than the alternatives. There are plenty of docs that will give the hand-holding that you are looking for, so nice you have made the decision.
If you know what you are doing there are arguments for other options, and data scientists have different tool chains. But I think we make it harder for new engineers by having five different ways of doing it, each with people arguing that actually their way is better.
And I remember custom build scripts with Cargo being a major pain at the time I used it, but the Cargo developers were able to sidestep major fundamental problems because they knew what didn't work with packaging in the decades before and managed to think about the design carefully enough from the start.
Rust is different because the choice of package manager is "simply" Cargo, not necessarily that Cargo itself is trivial to use.
which is the industry for which users are really the product. user's data in, profits out.
Personally I like Go. If someone wants to build my stuff, then I just say go here http://go.dev/dl and download Go, then set location to where the code is, and enter "go build". thats it. All languages should be that easy.
Go doesn't make a habit of interfacing with lots of legacy C,C++ and Fortran Code. There's probably some of that. But nothing like Numpy, Scipy, Tensorflow...
Also, Rust seems to indicate that being general purpose and interfacing with lots of legacy C (and some C++) aren't the issues.
Just about the only question I've fielded every so often in the past several months have been people who found their way to an old tutorial based on the old system and them being confused about what happens, and a quick link to the current docs seems to resolve all their problems.
These are the languages and package managers I have experience with:
- JavaScript and npm
- Python and pip/conda
- Kotlin and Gradle
- Scala and mill
- Common Lisp and quicklisp
- Racket and raco
- Raku and zef
- Lua and luarocks
- Elixir and mix
- Haxe and haxelib
- Rust and cargo
- Nim and nimble
- OCaml and opam
- Smalltalk and Monti/Meta-cello; also Visual Works parcel manager
- and a few others
Out of all these, cargo, opam, mix, and zef are the only ones that are mostly working as intended. They're still a pain, but they're predictable and honest about their limitations. Everything else is just bad, to the point where it's a waste of time to think about which is the biggest and which are a bit less of a clusterfuck. For some of these I had to resort to building a chroot env with a copy of my system inside to convince them to compile and link against correct libraries. If that's not a clusterfuck, I don't know what is...
Unless you were lucky to see something better than "mainstream state of the art" solutions, you won't realize how bad it actually is. At that point you start inventing yarns or poetries, adding layers on top of layers to paper over simplistic, hardcoded assumptions made by original authors 20-30 years ago, and go into denial when someone tells you that's not going to fix anything.
Somehow we ended up in a state where step one of doing anything new with python is to fire up an empty docker container. And I'm awfully tired of folks in the "python community" blaming the victims of the mess they made.
My recent example: last week, I was installing a new program that uses some python scripts during its operation. I could not get those scripts to run. I knew it was because of the usual python problem of the scripts not matching the version of python that was executing them, but figuring out how to fix that took me a full day.
Just to make Python work. Not even as a dev.
Wearing my user hat, this is a clusterfuck. There is no other language that I am exposed to that presents this sort of problem, and this is a very common issue with Python.
This is why I start to get sweaty any time that I'm using software that involves Python. It turns into a crapshoot and half the time, it's going to cost me a lot of time and stress.
> perpetuates the false image.
You can believe it's a false image if you like, but there are a whole lot of end user experiences that indicate it's very real.
I'm late to the party but AMA :)
Also, [even if it did], it installs binaries and doesn't compile.
You have already lost.
So if I had to chose one way for the next decade, I would bet on this.
But I will not pretend we can be sure about anything on this matter.
In fact, my hope is that we can improve the process tremendously by creating an unified way of bootstrapping Python for all OS. If this ever happens, the procedure would become mostly obsolete.
At the end of the week I will write about this, and the various experiments the community have done so far in that regard.
And it's often less about the package manager and more about the ecosystem: Can you find what you need? Is what you need stable enough? Does it break every few months with new versions of the runtime, either because of actual incompatibility (npm/node) or versioning shenanigans (nuget)?
But these are strictly package level issues, not package management.
The downside of node is the over reliance on flaky dependencies. The actual package management is good, especially compared to python.
It's not just "flakiness" of the package maintainers. The basic problem is that there is no single standard javascript package, and no package manager, especially npm can solve this.
That's not "being toxic", you just can't be everything to all people.
I spent about three hours trying to figure out how to setup python keyrings to work, to let me just get started using poetry. On a system I was ssh'ed I to. Gnome-keyring-daemom was up. I spent a while adding random pam rules suggested by archwiki in to inject more gnome-daemon stuff in my envs. Random gnome-keyring-unlock scripts, which quickly start talking about Clevis and tpm and fido 2-factor. Wading through hundreds of responses about Seahorse, a gui tool unsuitable for ssh. Many long miserable sad stories.
In the end I stumbled upon someone who suggested just nulling out & turning off keyring with some config to make it have a null provider. After this the poetry project just worked.
The tiny handful of deps this project has were already installed on my system, but poetry was also a task runner, instrumental for the usage of this single-file script.
There's been so many years of churn in the python world of tools. A fractal nesting doll of virtual-env, mkvirtualenv, & various offshoots. I hope some day there is a mature reasonable option folks generally find agreeable. Poetry eventually worked for me, but what a miserable gauntlet I had to walk, and the cries of so many who'd walked the path & utterly failed echoed out at me at every step.
For example, you can install oscclip to a temp dir with a dedicated venv and run it immediately:
$ pipx run --spec 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help
$ pipx install 'oscclip @ git+https://github.com/rumpelsepp/oscclip'
$ pipz runpkg 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help
$ pipz install 'oscclip @ git+https://github.com/rumpelsepp/oscclip'You don't need poetry for that. My single CLI packages still use a setup.py that I haven't touched in five years and was simple enough to write.
"There should be one-- and preferably only one --obvious way to do it.", unless it is how to setup your environment.
I only use Python every few months, and it is always a struggle.
In comparison, "cargo build" works 98% of the time just after "git checkout"
conda solves it by packaging EVERYTHING, giving you atrocious 30GB environments, pip doesn't solve it at all and none of the challengers really have much to offer (in my opinion).
Disclaimer: I was using Fedora which has Python 3.11, using Fish which is clearly non-standard and I’m a sysadmin not a Python dev.
`${my-venv-path}/bin/python`
`${my-venv-path}/bin/${whatever-binary-you-need}`
`%my-venv-path%\Scripts\%whatever-binary-you-need%` (because Windows...)
When there's a new python version I'm interested in, I install it via Homebrew and update my zshrc to clobber everything else via $PATH. All my scripts and tools are broken? Just reinstall the packages globally. Whatever.
Since the big 3.x transition, it's pretty rare for forwards-compatibility to break (IME), and if something does, I can just try running prior python3x binaries until I find the last version that worked.
It's hideous, but honestly the least stressful way I've found to date.
If you "simply" (yes yes I know) download the python version you want directly and compile/install in a local folder, and use that with venv as a way to manage individual project dependencies, all problems go away.
Python packaging is fine for small local dev; problems arise in distribution, rollouts of upgrades and ensuring your apps work in the zoo of local setups your users have
How to transition a bazillion packages though I do not know.
It saddens me to say, but imports are indeed a complicated topic on Python. The PYTHONPATH is not something people know about, and once they do, it's not intuitive.
The weird handling of namespaces or relative paths doesn't help.
>> I think you missed the point.
Maybe I did, but I've been using Docker as version management for pretty much every technology I employ for five or six years. Prior to that I sparsely used things like rbenv and virtualenv and I actually thought it was super dangerous and unreliable. Maybe it's gotten better in recent years, and certainly people who write python and ruby every day are going to know more about this than I do.
I don't install anything on my computer if I can just use Docker for it. OK, I do have go:latest, but I use docker images for various projects that might be on any version of go from 1.8 to 1.20. Your website still runs on PHP5.3? I can help you (I won't, but I could totally run it locally!).
Reasons I like docker better:
1. Any scripts or configs can explicitly refer to the version number. No guessing or assuming.
2. Our whole team uses the same version.
3. Only one dependency: docker.
Granted I'm more of a sysadmin than a developer and I'm sure that biases apply.
If you have a Docker image that builds with "pip install whatever" and this library gets updated with a breaking change, you won't be able to rebuild the image without changing the dependency or the code itself, for example.
On the most tightly managed lab machines, which are all in lockstep on a fixed configuration (latest Ubuntu LTS with a updated image pushed annually), we can provide a consistent Python setup (e.g. Python 3.10 and a fixed set of C-based modules like psycopg2). However, our staff and PhD desktops and laptops are more diverse - with the OS often only being upgraded when the distro is going out of support, they could be running n, n-1 or n-2. That, most likely, means three different Python versions.
We could use pyenv to let people install their own preferred version. Installing with pyenv requires building from source (slow on some of our oldest machines). This also means installing the Python build deps, which is fine for our departmental machines but not possible on the HPC cluster (operated by a different business unit) or the Physics shared servers. It's also less than ideal for our servers where students deploy their projects (where we want to minimise the packages installed, like the build-essentials meta package).
It's also a massive stumbling block for less experienced students with their own laptops which could be running any distro, of any age. Many CS101 or engineering/business/humanities students taking a programming class, who have never programmed before, would really struggle.
So, classes might tend towards teaching lowest common denominator Python (i.e. the oldest conceivable version a student might have installed on their machine).
Sure, we have in-person and remote lab machines students can use - but it's not always convenient (especially for the data science / ML students running Jupyter notebooks with their own GPU).
There are workarounds, but they all have serious downsides.
Compared with Node.js and Go, where users can just download the appropriate package and unzip/untar the runtime or compiler version of their choice, deploying the Python runtime has enormous friction (especially for less experienced users). This has the bonus of simplifying deployments elsewhere in our infrastructure (CI/CD, containers, etc).
And while we all complain about node_modules, Python venvs not being trivially relocatable is another huge frustration.
We've used Anaconda, but that comes with its own issues (most recently, finding that it ships with its own gio/gvfs binaries and libraries which fail to mount our CIFS DFS shares - causing confusion for users running `gio mount` from within a conda environment).
We also haven't figured out how to incentivize maintaining backward-compatibility. 99.9% of the time when some library updates, and stops working with language version X, it's using some hot new feature of the language. Usually just because the library author thought it would be cool or more elegant. The entire software world needs to update now, because someone left-padded us for elegance.
I'm just sticking all dev projects into a separate LXC and calling it a day. Don't want to deal with all the various separation models the various languages and package managers cooked up
https://gist.github.com/tom-pollak/8326cb9b9989e0326f0d2e19f...