Anyway, has Poetry caught up to the dependency resolution provided by the pipenv lock command yet? Last time I tried it (~6 months ago), it couldn't produce working environments for my requirements.
BTW, for anyone who wants to educate themselves, read the issue description here and follow the links: https://github.com/pypa/pip/issues/988
Was the problem with dependency resolution you had similar to what Poetry describes in it README? Or are there other problems the README fails to mention?
The description they make got me interested, but up to now all alternatives I tried to build moderately complex python projects failed to deliver on their promises, so definitely interested in what you have to say.
[1] https://github.com/sdispater/poetry/blob/master/README.md#de...
Been meaning to look into poetry. Not a huge fan of TOML, but hopefully all the tooling supports pyproject.toml now (I know black does, but not sure about flake8, isort, pylint, pytest, coverage). I know there's still some hacks required for tox + poetry though.
I found the tox support a bit hacky, but not bad, and got 3.5 to 3.8 testing just fine.
Poetry does make the publishing process pretty seamless for the simple case. If you're starting a new library, poetry is great.
I think poetry needs a "fuzz option" to test random valid solutions to the version constraints, to check if the older versions are really valid.
What config language would you prefer to use instead?
I've tried similar configurations with poetry and it doesn't seem to have a problem with them.
It is important to apply pipenv as intended; I only use `pipenv lock` and `pipenv sync`, and edit Pipfile manually.
The problem with other commands such as `pipenv install` is that they don't apply the holistic dependency resolution as `pipenv lock` does.
iirc Poetry does have cross-platform lock files and a coherent story. I'm just blocked on some PRs due to the original author getting busy and not yet having delegated to lieutenants.
But it doesn't handle platform-specific dependencies correctly.
For instance, `doit` has a watch feature, and it will depend on pyinotify on linux and macfsevents on Mac.
If I run pipenv lock on the Mac, it correctly generates this:
"macfsevents": {
"hashes": [
"sha256:hexhexhex..."
],
"markers": "sys_platform == 'darwin'",
"version": "==0.8.1"
},
[1]: https://github.com/pydoit/doit/blob/660d3a2e75dc9d861e5a8916...
$ pip freeze |wc -l
204- vanilla virtualenv: I don't even bother with the wrapper most of the time.
- vanilla setup.py/setup.cfg: Its just really not that bad. Forget about project.toml or whatever the next big thing is.
- pip-tools: ditching pipenv and using this for pinning app requirements, has made my life so much simpler.
export VIRTUAL_ENV=${XDG_CACHE_HOME:-~/.cache}/virtualenvs/$(basename $PWD)
export PATH="${VIRTUAL_ENV}/bin:$PATH"
python3 -m venv $VIRTUAL_ENV
In all the years I never had any need to use the real activate method. Just running Python and scripts directory from the bin/ in the virtualenv is enough to run 'inside the env'.
-e git+https://github.com/...
git+https://github.com/...
Also can you pin your dependencies to a specific hash? Last I checked you cannot. A major advantage of pipenv/Poetry is that the lockfile protects you against compromise of the package manager; if someone swaps in a new binary for yourdep==1.2.3, the lockfile will fail to validate the hash and you'll get an error.
The one large problem of those are forgetting about activating a venv, so I get I'll spend some time learning how to use direnv from aequitas advice.
I like Poetry a lot; I hope platforms like Heroku will start supporting it soon.
That's a shame it is not in the project webpage, though. This is the first information I searched, without success.
Using --user here, no per-project virtualenv, used to have a virtualenv in ~/.env, where i keep everything up to date and have many -e installs from ~/src. For production deployments: --user in containers.
This means all my projects have to work with the same versions of dependencies.
This means all my dependencies have to work with the same versions of dependencies.
All my dependencies have to be up to date: all versions must be latest, or next to-be-released.
When don't support new versions of other dependencies: I deploy a fork and open a PR.
At such, I'm not spending time trying to make obsolete versions work, rather, I'm spending time making new versions work together.
My perception is that this strategy offers better ROI than all others which I have tried, not only for me, but for the whole ecosystem. That's also how I made my Python 2 to 3 transition, and have been using 100% Python 3 for a while (once you've ported one big software, porting others and their smaller dependencies is easy).
I'm extremely happy about this strategy and my life has just been better since I started working this way, and highly recommend it.
For my own (countless) packages: I use setupmeta which simplifies auto-updates (I think it succeeds what pbr tried to achieve).
This means all my dependencies have to work with the same versions of dependencies.
Then why not just install everything globally?
For people working outside of scientific Python: conda is a package and env manager maintained by a private company that's become the go-to because it's really good at handling binary dependencies.
As far as my understanding goes, conda focuses on providing virtual environments for interactive use, but to build and distribute a package on conda forge, you still need to rely on setuptools/distutils.
My impression here is that Poetry aims more at _replacing_ the pip + setuptools toolchain. Users of your package could still install it using conda, if relevant. It seems a bit limited in what it can do at the build step, unfortunately, so it is not a replacement yet.
Coming from the Java world, I pass my days dreaming of a "maven for python", and this project definitely goes into that direction. I will definitely keep it on my radar.
conda build recipes are essentially just shell scripts that conda runs in a sandbox, taking care of library paths and so on. You could, if you really wanted to, develop a poetry based Python package and then have your conda build script use/run poetry to build and package the python package, and then add whatever lines are necessary to the build recipe to make that the package that gets installed by conda.
I wish he'd run the project a bit better. Other than that, it's stellar software.
That said, "ejecting" a plain requirements.txt is possible on the alpha branch via `poetry export`
The biggest problem with Python packages isn't a dependency manager, it's that Python developers don't reuse and extend existing projects. They just keep churning out new similar projects with different features. All of the most commonly used Python packages are basically one-offs that use a bunch of other one-offs. This creates a cycle of reinventing the wheel and friction in ramping up on existing technology, due to all the extra integration work and split development, and later lift-and-shift into completely different dependencies that implement the same thing. PyPI is awash with duplicate abandoned one-offs to the point that just trying to just find a useful module to do what you want can take a long time.
I wouldn't call that "easy", but OK.
# Makefile
venv:
python3 -m virtualenv --version 1>/dev/null 2>/dev/null || \
( echo "Please install virtualenv (python3 -m pip install virtualenv wheel setuptools)" && false )
[ -d venv.d ] || python3 -m virtualenv -p python3 venv.d
./venv.d/bin/pip install -r requirements.txt
$ make venv
# Running your project without setup.py
$ ./venv.d/bin/python3 my-program.py
# Running your project with setup.py
$ ./venv.d/bin/pip install .
$ ./venv.d/bin/my-program.py
I hope the rest of the ecosystem can catch up quickly.
Tox and pip and and pex need full support for PEP 517/518.
I still wish poetry would support setup.cfg in parallel. It has been working for 2 years, is compatible with pip, setuptools and the whole legacy stack, and most fields are standards and documented.
I would like to see tox support `install_commands` (https://github.com/tox-dev/tox/issues/715)
I would like to see pex support the `build-system` feature. (https://github.com/pantsbuild/pex/issues/660)
For pip, I'm not sure what's involved in getting everything to work generically. This is probably not simple. It has the consequence that when we don't have control over the install command, we end up needing hacks like using `extras` for ReadTheDocs (e.g. https://github.com/readthedocs/readthedocs.org/issues/4912#i...)
Venv doesn't do dependency management, it just provides an isolated python environment. By default, poetry uses venv for environment isolation, but also does dependency management.
yarn came out great but these days I switched back to npm for the sake of simplicity of my workflow, especially npm absorbed yarn's good features.
same thing happens to parcel/webpack, the neat new tool beats the old players, but then the old dominant one catches up by taking in good stuff from its smaller competitors, quickly.
With poetry+pyproject.toml, I have one file that is flat, simple, and readable. @sdispater has done incredible work on this project, and I hope they get more resources for development.
Quick question since you seem to know npm very well, is there a better solution than this to automate npm package publishing nowadays ?
sed -i "s/GIT_TAG/${CI_COMMIT_REF_NAME/v/}/" package.json
(I don't have this issue with Python's setupmeta)
Python makes huge sacrifices in readability and efficiency for a hypothetical win in writeability. It's also fundamentally at odds with the multicore world we're living in an will continue to live in, an issue that many people have tried to fix and failed at.
I can't count the number of times I've had to spend my entire night reading through an older Python service without type annotations, desperately trying to understand the types of inputs and outputs. It's just a mess, and the bias of programmers to believe that they are writing readable code (when in reality it's only readable to them) exacerbates the use of names that provide little context to solve the problem.
Python is awful, and it's an uphill battle to fix it. Asyncio can solve some performance issues, but negates the benefit of Python's simplicity by destroying the sequential mental model that Programmers can easily grok. It also requires a complete rewrite of libraries and a split between non-asyncio land and asyncio land.
Type annotations can make Python more readable and thus more maintainable, but gradual typing still leaves lots of unanswered questions when you're interfacing with untyped code.
Python is really not good unless you are using it to prototype or build something on your own. It's led to a world of slow, buggy software that is impossible to rewrite. It's downsides are easy to measure, but its benefits are difficult to quantify.
One solution at the moment is to run 'pip freeze' and put that in as your requirements file, but that very much feels like an 'and now I have a different problem' solution.
I can't be too mad at pipenv, but all in all poetry is a better experience.
I see though that it only supports pure python for building packages, does that mean that it doesn't build if you are dependent on compiled libraries?
Is there also a plan to add some of the functionality of bundling tools such as web-pack into this build phase? like automated css optimization, image compression... Could be handy for some django/flask projects.