And then I got to re-learn how very bad Poetry and python dependency management is. I’m trying to update pyyaml to 6.0.1 and Poetry systematically downloaded every single major, minor, patch version of another dependency in trying to find which would fit the version constraints. Took half an hour.
Sounds like fundamental design issue more than anything.
The only semi-sane workaround I know of is to use pip's `--only-binary` option to prevent any automatic builds at package installation time. Then you usually will also need your own package server for storing precompiled wheels for those third-party-dependencies that don't published compiled packages on pypi. This way you build the packages using some random tool version only once, so if it works the first time it will keep working.
In other language ecosystems were the package management isn't built on a tower of shit, this problem doesn't exist in the first place.
Great news! Better do that then have to do it manually
But yes it sucks, still Poetry is much better than Pip (where it just gives up and tell you to go deal with the issue yourself!)
Now why do people care about Cython and especially in things like dependencies I don't understand. awscli shoud be an independent package! "oh but it is faster" are you counting the time it takes to fix things when they break because you are depending on a fragile thing?
If one of the build deps has no wheel then you're fucked, though. At some level you have to rely on a toolchain.
I was pretty annoyed to discover that it had a separate build-time dependency on cython that you can't control from your requirements.txt.
[1] https://github.com/spack/spack/ [2] https://github.com/spack/spack/pull/35995
The license_file parameter is deprecated, use license_files instead.
By 2023-Oct-30, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
Didn't they move away exactly because of issues like this? As I understand it, aws-cli v2 is still written in Python but ships everything it needs self-contained so it can avoid Python dependency hell[1]
cat -n shell.nix
{ pkgs ? import <nixpkgs> {}
}:
pkgs.mkShell {
name = "dev-env";
buildInputs = with pkgs; [
awscli2
];
}
I have some humongous (for me) dict lookups I am doing on strings, counting in ints, and if I could 2x or 3x faster without having to recode I'd love it. The Cython web is directed at people willing to recode, I am ultimately willing but I wish there was some low hanging fruit evidence: is this even plausibly going to get faster or is dict() walk cost just "it is what it is" ?
Oddly, it looks like across pickle dump/load I get some improvement. Does pickle restoring consume less memory than raw-made?
I wouldn't expect a full 2x in every case. But it can also be very significant in cases where Cython deduces the type of local variables.
The way I look at Cython is that it is a tool that allows me to write C code using python syntax. The biggest gains you get is when you use annotation to specify use simple (C) types with their drawbacks (like overflow etc) and or python features like exceptions, GIL etc
Having used swig to create Python bindings for C++ code over 10 years ago, what’s the recommended way to do this in 2023? There’s still swig, there’s Cython, pybind11 and a few others. I do know swig and how complicated it can get with footguns abound when things grow more complex.
Is Cython the way to go? How does it hold up to the alternatives? Google search gives many articles on the topic, but many typical SEO optimized low-value, and those that do show a bit of depth, focus on the basic mechanics, not really on things hold up for larger projects…
If you just want the ability to provide a Python interface to a C/Cpp library PyBind11 will get you there in fewer LoC than Cython. Nanobind is an even lighter weight option.
I’ve heard Swig is a pain to use.
From the user POV, the best bindings I’ve seen were wrappers with a Python API that calls C++ using Cython.
Other projets not pinning their dependencies is not their fault!
1. Memory unsafety. It's still C or C++ in the end, the more you shift your code in that direction the easier it is to screw up.
2. Two compiler passes: first Cython->C, then C->machine code. This means some errors only get caught in second pass, when it's much harder to match back to the original code. Extra bad when using C++. Perhaps Cython 3 made this better, but it's a very hard problem to solve.
3. Lack of tooling. IDE support, linting, autoformatting... it's all much less extensive than alternatives.
4. Python only. Polars is written in Rust, so you can use it in Rust and Python, and there's work on JavaScript and R bindings. Large Cython code bases are Python only, which makes them less useful.
Long version: https://pythonspeed.com/articles/cython-limitations/
There seem to be a lot of cool things in this. For example, semantics for division, power operator, print, classes, types and subscripting are now identical with Python 3. https://cython.readthedocs.io/en/latest/src/changes.html#imp...
Improved interaction with numpy by moving that part of the code into numpy itself: https://cython.readthedocs.io/en/latest/src/changes.html#int...
Improved support for const and volatile in C, and improved support for C++ expressions like std::move https://cython.readthedocs.io/en/latest/src/changes.html#id6
I love how the changelog also shows the bugfixes and improvements that each of these changes enabled. Hats off to the team behind this release.
> Since Cython 3.0.0 started development, CPython 3.8-3.11 were released. All these are supported in Cython, including experimental support for the in-development CPython 3.12
Nevermind the foo broke bar comments here.