I don't understand why the Python leadership hasn't shown stronger... leadership... in which tools they recommend.
All that could have been prevented by the Python folks who insisted on deprecating a builtin module. They could have jumped in and help migrate the popular Python packages away from distutils long ago, to set an example. But nope, they really like to keep things messy.
They took away cgi… which I was of course using.
And crypt.
They don't care about breaking things. People would not migrate to python4 so they just break everything in python3 :D
If you're putting something like a web app or something else that will be bundled and distributed as a unit, then you're probably best off with something like Poetry, PDM, or pip-tools - you have a lock file for deterministic dependencies, most of your dependencies will be pure python wheels, and you only really need to test things once. On the other hand, if you're developing a library, you'll need to test against multiple versions of Python, and ideally multiple versions of some of your larger dependencies, to ensure that your library will work for as many users as possible. You'll also need to be able to build and package wheels. Alternatively, you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them - specifically, so that they work with the GPUs and other hardware you have available. And there's still the group of people writing mainly scripts for server maintenance or other tasks, who want to be able to easily install dependencies and keep those dependencies up to date for security reasons, with the minimum number of breaking changes.
Right now, there are different tools, packaging systems, etc catering to each of these groups, and so building the One Ring of Python package management is going to involve (a) solving all of these problems, and (b) convincing all these groups of people that your general solution is better than their niche-specific solution. That's certainly not easy, I don't even know if it's all that possible.
I do think that working from the ground up (i.e building the individual components like the package metadata file, or the pypackages folder experiment) seems to be working well, in that tools seem to be coalescing around these options and finding the best ways to use them, which is all work that might hopefully feed into new official tooling. But we'll see.
* Pure python -- Easy, use one of the declarative ones.
* Python + Standalone C -- not too bad, use the build tool.
* Python + external (potentially distro supplied) C libraries -- Using setup.py, customized, and different for each project.
Honest question from a web developer who sometimes has to work with Python — don't containers solve exactly this?
But yeah, I agree. This is something that golang nails. They supply all the options out of the box.
That's what Python packaging feels like. At least videogames are fun.
Replace "the game" with "the small sphere of specialisation" and this applies to learning a great many things which don't generalise, many of which don't make you money either.
Your friend doesn't have a critique of video games, he has just discovered relaxation and/or hobbies.
Competitive team games are the best examples. Playing them well requires internalizing the concepts of probability, teamwork, mindset, efficiency, dealing with failure, etc.
Those skills come quite in handy because life is a game. It's especially obvious when looking at human-created systems like capitalism and jobs.
"Completely wrong" is too harsh. What I said only holds true when a player puts in the effort to get good at a game. Mindlessly playing Cookie Clicker while commuting will not yield many benefits.
If you play some useless¹ game for the majority of your days, you can have multiple reasons for this. Maybe it keeps you occupied and thinking of the problems in your life? But after a certain degree it certainly does not "entertain" you anymore: You are doing it because doing anything else that you can think of feels worse. And a few years after not a lot will be left of it, unless you played with your friends or there was something in it for you beyond just pushing the time forward.
If I am playing my instrument for hours, I have at least improved at expressing my feelings with my instrument while having a generally good and relaxing time. And after years of doing it I can do it well enough to play on concerts without feeling afraid.
I know to many people who are so afraid of their own thoughts, they will obsessively "entertain" themselves dueinf all wake hours, and the majority of the time they are not enjoying it. This is bad. We only have one life. You don't get a price for having some the most, but at least do something that is meaningful.
¹ a game that you yourself don't even love that much
To harmonize these ideas, I argue to the kids that the "entertainment" choices are like spice on the food. We want enough to enjoy the flavor in question, but not so much that the usage becomes expensive or overpowers the dish.
Why not just improve upon pip? I don't know, just have pip use toml and give it different flags or auto detect things? Was a whole new ecosystem of tools needed?
I look at decisions like this and I know a Python 4.0 is on the way, just like the 2->3 jump because why not right?
Any language that updates it's syntax or toolset in a way that is impossible to have backwards compatibility is an irresponsible toy language, as awesome and powerful as it may be, the developers are still toying and tinkering with it with no regard to real world impact of their decisions.
Why can't python3.12 have a flag like --std=2.7.18 for example, like gcc? If the devs don't have enough resources, I would live to see a donate page I can help out with.
We are at a point where to deploy python you can't use python but you need shell scripts to figure out versions, venvs, pyev, pipx, poetry, etc... and reliably fail of course and every time you have to troubleshoot the problem. This is a failure in software engineering, new grads should be taught of python and similar languages and the lack of planning and organization and resulting cascading chaos as examples of how not to design the user experience of any piece of software.
Sorry if I exaggerated a bit anywhere, it's difficult to pretend all the frustrations and crying out "why???" when using python don't exist. But at the same time, it is still my #1 go to language for most use cases because the language itself is just fabulous!
God forbid you have to upgrade a base python version and reopen the dusty tomes of python dependency hell.
They are bathed regularly in the blood of a thousand virgins and anything but dusty
I was sick of having 10 different files in the root of my project to config the different tooling that the project used.
I got so sick of it at one point, I went ahead and wrote the code to support the pyproject.toml file for mypy because it was my last holdout.
It's not great, but it's also not terrible.
E.g. https://github.com/cbarrick/efiboot/blob/2ca46a7c27c837adf23...
It seems like with a tenth of the effort of this blog rant the author could have written a flowchart or best practices.
Luckily for 99% of Python users, we only need to install libraries and not package them...
https://blog.isquaredsoftware.com/2023/08/esm-modernization-...
I _liked_ Gregory's post. I _felt_ all of the frustration, and the confusion over lack of good docs, and the plethora of competing tools and formats. It resonated deeply for me. I haven't touched Python in years, but I could see exactly how all these pain points were happening thanks to the writing style and the explanations. The goal is to share "here's what I tried", "here's what seems wrong or broken", "here's the pitfalls I ran into", and "here's the _frustration_ I'm feeling at how messed up this all is". And it succeeded.
(And fwiw, I've had a lot of other folks in the JS ecosystem express similar thoughts as thanks for my post. It's a limited target audience, most people won't care, but the folks who _do_ have to deal with these kinds of problems understand and appreciate the details and the effort involved.)
********************************************************************************
The license_file parameter is deprecated, use license_files instead.
By 2023-Oct-30, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
********************************************************************************
https://github.com/pypa/setuptools/commit/3544de73b3662a27fa...
This `license_file` change is the epitome of unnecessary breakage.
Having the mess of different blog posts and documentation sources saying different stuff is far from ideal though. If you can't sum up the process in a clear concise way you are far from done.
I don't understand... Poetry solves this too.
SHARE YOUR WORK
Publish
Make your work known by publishing it to PyPI
You can also publish on private repositoriesI was sad to see setuptools officially deprecated, because it looks like another way in which Python packaging is being red-taped away for a non-expert. If someone like the OP (who has 10+ years programming Python) had to do so much for what appears to be a zstd CFFI/Rust wrapper, where does that leave the rest of us?
Here's a python package of mine that uses setup.py: https://github.com/ahgamut/cliquematch/blob/master/setup.py which I have not upgraded to the new tool(s) yet. I think I will need to upgrade it soon. If anyone has suggestions for a tool that will _fully replace_ setup.py, I would like to see tutorials with the following examples:
1. How would I build a package that has pure-Python files and data files? With setuptools I would use maybe MANIFEST.in or package_dir.
2. How would I build a package that has a CPython extension accessed via cffi? (this post points to the answer)
3. How would I build a package that has a CPython extension _without_ cffi, that just wraps some small C code I wrote with CPython's API? What about an extension that uses PyBind11? What about an extension that uses Rust?
4. How would I build a package that requires a "system" package like libblas-dev? Can something like numpy be built optimally without ever writing setup.py? What would a config for that look like? Last I remember numpy used their own patch of distutils to build, I wonder what it is now.
https://learn.scientific-python.org/development/guides/packa...
As a TLDR, you have many options 3rd party build tool (aka build backends). Each build tool have different *static* ways to specify compile options that is native to the language or generic (e.g., CMakeList,s Cargo.toml, 3rd party YAML. When it comes to dynamically specifying your extensions, setuptools is still the only option.
Having said that, the author also sounds like he's ranting a bit. He seems to insist in finding specifically how to work the way setup.py used to, but without setup.py, instead of just learning how to use pyproject.toml. While learning the new way of doing something, how it replaces the old way is usually self-evident. The (official) tutorial he eventually lands on (https://packaging.python.org/en/latest/tutorials/packaging-p...) is actually a pretty good primer. Without previously knowing what hatchling, build, twine, or even pyproject.toml was, I was able to quickly understand their purpose. From clicking a few other links on the side bar, I understood that packaging is done with tools that present a frontend and interact with a backend. Sometimes a tooling set provides both. Hatch seems to be the frontend of one such project, while Hatchling is the backend.
That tutorial you linked to does not describe how to handle the issue the author faced in building a Python binary extension. It doesn't even describe how to build a C/C++/whatever extension.
At the bottom of the page the text "Read about Packaging binary extensions" points to https://packaging.python.org/en/latest/guides/packaging-bina... which starts "Page Status: Incomplete" and "Last Reviewed: 2013-12-08", and contains a number of "FIXME" sections.
(Parts have been updated during the decade, but the contents of the page do not instill confidence that it's up-to-date.)
That documentation still doesn't show how to configure the extension, instead linking to https://docs.python.org/3/extending/extending.html . That in turn links to https://docs.python.org/3/extending/building.html which links to https://setuptools.pypa.io/en/latest/setuptools.html which takes you to https://setuptools.pypa.io/en/latest/userguide/ext_modules.h... showing how to configure your setup.py.
However, unlike using setup.py directly, where you can hack setup.py's argv, that documentation does not show any way to specify compile-time parameters, "like --system-zstd and --rust-backend as a way to influence the build".
Which is what the author wants to do.
I knew it was “the old” way, but didn’t realize it was abandoned.
Getting your code packaged seems way harder than developing your code.
Modern way of building a package is to use a build frontend. Most notably is the PyPA's build package.
python -m build .
I did not get halfway through this before I could start to feel the hairs of my neck start to stand on end. The numerous blind alleys. The promising leads that aren't. The official documentation that contains links to other documents that contradict the first and so I have to try and also piece together some temporal state to make any sense of any of it. Some days it seems I am actually some kind of an information archeologist, piecing together the detritus of overlapping civilizations.
One thing the author might want to try is writing their own "build-backend". You can specify your own script (even use setup.py) and that will be the target of python -m build or pip wheel or presumably whatever build-frontend you use.
# pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setup" # import setup.py as the build-module
backend-path = ["."]
def build_sdist(sdist_directory, config_settings):
...
def build_wheel(wheel_directory, config_settings, metadata_directory):
...
example:
$ python -m build --config-setting=foo=bar --config-setting=can-spam
# will call
>>> build_sdist("the/dist/dir", {"foo": "bar", "can": "spam"})
from setuptools.build_meta import build_sdist as setuptools_build_sdist
def build_sdist(sdist_directory, config_settings):
# ... code-gen and copy files to source ...
# this will call setup.py::setup, to make things extra confusing
return setuptools_build_sdist(sdist_directory, config_settings)
Still overall pretty fiddly/under-documented and a shame there seems to be a push for more dependencies rather than encouraging users to build their own solutions.
More info in PEP 517: https://peps.python.org/pep-0517/
I open https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html
in my browser and see a 4,000+ word blog post.
Oof. Do I really want/need to read this?
-----
Kidding aside, good content and a great reminder to think before blindly upgrading—at least until the kinks and details are worked through
The blatant mess that ensued in the meantime (i.e. last few years) proves me right, imho.