Not sure where I would stand if I fully investigated it tho.
[0] https://github.com/python-poetry/poetry/issues/6409#issuecom...
Due to Poetry's architecture, it can't satisfy all three at the same time:
- the platonic ideal of build isolation and lockfiles
- installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects
- dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere
This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles.
Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. They have something that will not correctly install anything that is dependent itself on PyTorch, which is useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry.
Here's the facts of the matter:
- 1.2M requirements.txts https://github.com/search?q=path%3A%2F%5Erequirements.txt%24...
- 664k setup.pys https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type...
- setup.pys that reference requirements.txts https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requ... 67.1k
- only 30k pyproject.toml specified with dependencies https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2...
2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.
What does it mean, build isolation? And how does PyTorch violate that? Or why can't we install PyTorch with build isolation?
What is the issue with lockfiles?
I can understand there might be issues if you need to install a custom accelerator-specific version of PyTorch (although also I don't fully understand what the issues are about this). However, what's with just the standard version, which is suitable for most people, as it comes with CUDA support?
What do you mean by non-interactively? pip install is non-interactively, or not?
What issues are there with the dependencies for PyTorch? Why are they not compatible with the dependencies expressed elsewhere?
How are dependencies defined by requirements.txt different than dependencies defined by pyproject.toml or by setup.py?
If your answer is RTFM, maybe you can point me to some resources. I read a bit through Poetry documentation, but I still don't really understand those issues.
"Build isolation" is all about limiting what is available at build time (this generally conflicts with inspecting the environment/system), and is vaguely (but no where near sufficient) for reproducible builds (see https://reproducible-builds.org/, which came from linux distros). Lockfiles are similarly an issue because they try to assume they can cover all cases (which they cannot, see e.g. various discussions about sdists for the lockfile PEP). Both these conflict with trying to work out what works for the current system (which is what pytorch tries to do because there are so many different options), or with trying to control what is discovered when trying to do the build on one system but run on another.
Frankly setup.py is a big hack and we should strive to do better.
Poetry is great if you're developing the sort of software that the poetry devs priorities, but since poetry is only targeting a subset of the python community it will never be a good default tool for all python development, and thus the fragmentation continues.
