https://github.com/charliermarsh/ruff
It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.
relevant section from my pyproject.toml
[tool.ruff]
line-length = 88
# pyflakes, pycodestyle, isort
select = ["F", "E", "W", "I001"]What do you mean by this? Are you indenting Python with tabs?
Though their `v0.0.X` versioning is very funny to me (https://0ver.org/).
replaced both flake8 and isort across all my projects
I would nitpick this. You build images not containers and since files are not copied by default there is more nuance here that the .dockerignore file makes builds faster by not including them in the build context.
That does ultimately prevent COPY directives from using them but it is these sorts of brief, slightly inaccurate summaries that mislead folks as they build understanding.
> slightly inaccurate Not entirely, I'm not sure the author even wanted to stress on this in the article. People won't learn docker from a python article about the same.
I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code.
Black shouldn't break anything (and hasn't broken anything for me in the years I used it) but in the unlikely case it does it, there's still pytests/unittests after that that should catch problems...
Even while it won’t break anything you want CI to be your safety net, flagging a local setup as being wrong is more valuable than magically autocorrecting it.
CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such.
That going too far unless you define code to be a subset of the files checked into the repository and simply define any file that's touched in an automated manner to be not code
There are a lot of useful automations that can be part of the CI/CD pipeline, such as increasing a version number, generating a changelog, creating new deployment configuration etc
They don't have to be part of it and it's possible to work around it/don't commit... But that comes with it's own challenges and issues
I liked black, though I was never satisfied with the fact that there was no way to normalize quotes to be single quotes: '. Shift keys are hard on your hands, so avoiding " makes a lot of sense to me. But there's the -S option that simply doesn't normalize quotes so it has never been a real issue.
However, this new project has a lot of typer functions with fairly long parameter lists (which correspond to command line arguments so they can't be broken up).
black reformats these into these weird blocks of uneven code that are very hard to read, particularly if you have comments.
Everyone is a fan of black; no one liked the result. :-/
I have a key in my editor to blacken individual files, but we don't have it as part of our CI. Perhaps next project again.
100% this. I also let Black auto-format code in the CI and commit these formats.
A lot of developers, intentionally or not, don't have commit hooks properly setup. If Black doesn't change the code in CI they need to spend another cycle manually fixing the issues that Black could have just fixed for them.
You're saying that there's a risk that Black could break your code when formatting? Well, so could developers and I'd trust a machine to be less error-prone.
There is nothing more frustrating than coming back from a coffee break only to find out that you have to rerun your CI check because of a trivial formatting issue.
Let black format code before it is checked in. Code should not be reformatted for CI or production, and bad formatting should either ALWAYS throw errors (no known defects allowed) or NEVER throw errors (if it passes tests & runs ship it). Consistency is the key.
Also, mypy has gotten really good in recent years and I can vouch that on projects that have typing I catch bugs much much sooner. Previously I would only catch bugs when unit testing, now they are much more commonly type errors.
The other thing typing does is allow for refactoring code. If anything, high code quality relates to the ability to refactor code confidently and typing helps this. Therefore I would put it at the top of the list above all the tooling presented (exception I agree with ci/cd)
There's zero harm in using list in private interfaces: I know I'm the only one passing the value, I know it is always a list.
As an argument type, Iterable is compatible with list, so it's benefits are minimal (with rare exceptions).
Lists are easier to inspect in a debugging session.
Iterable can be useful as return type, because it limits the interface.
Iterable is useful if you are actually making use of generators because of memory implications, but in this case you already know to use it, because your interfaces are incompatible with lists.
I can count on fingers of my hands when using Iterable instead of list actually made a difference.
Iterable is not compatible with list, but list is compatible with iterable. As the more general type, Iterable is better as an argument type unless you have a reason to force consumers to use lists. Even in private interfaces, I tend to prefer it, because I often end up wanting to pass something constructed on the fly, and creating an extra list for that rather than using a genexp just seems wasteful.
`list` might be but `List` isn't. Are you not defining the type of the contents of the list?
https://docs.python.org/3/library/typing.html#typing.List
> Deprecated since version 3.9: builtins.list now supports subscripting ([]). See PEP 585 and Generic Alias Type.
No. What allows you confident refactoring code are automated tests. I honestly can't understand why people are so obsessed about types, especially in languages like Python or Javascript.
By depending on interfaces/abstractions instead of specific cases you can refactor the interface and not break clients. It's very difficult to do this unless you have types.
This is something that Go is really good at and encourages but can be done with python/js on top of their type systems.
Types in Python feel like an added layer of confidence that my code is structured the way I expect it to be. PyCharm frequently catches incorrect argument types and other mistakes I've made while coding that would likely result in more time spent debugging. If you don't use any tools that leverage types you won't see any benefit.
It's a very powerful sanity check that lets me write correct code faster, avoiding stupid bugs that the unit tests will also, eventually, find.
And, to me, reading the code is much much nicer. Types provide additional context to what's going on, at first glance, so I don't have to try to guess what something is, based on its name:
results: list[SomeAPIResult] = some_api.get_results()
It's probably just a bad example, but in case it isn't:
Sounds like you ended up at the same place. You went from guessing what is some_api.get_results(), based on it's name, to guessing what is SomeAPIResult, also based on it's name.
If some_api is your library, then you could have just added type hints to get_results() and let type inference do it's job.
If it's a third party library, then using your custom SomeAPIResult means that code is becoming alien to other engineers that worked with that library in the past. It might be worth it, but it's definitely controversial. You probably should've done it with stubs anyway.
Typing facilitates automated testing; e.g., hypothesis can infer test strategies for type-annotated code.
[0]: https://github.com/agronholm/typeguard/
[1]: https://typeguard.readthedocs.io/en/latest/userguide.html#us...
These statements contradict themselves? List is too specific, and Sequence[item] is preferred. Sometimes you are dealing with a tuple, or a generator, and so it makes more sense to annotate that it is a generic iterable versus a concrete list.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Yet the Python type-annotation ecosystem was strongly oriented around nominal typing (i.e., caring that something is exactly a list) from the beginning.
I'm saying that this quote is a straw man and that contrary to what is claimed in the quote, instead, the ecosystem would go with/recommend Iterable[Item] or Sequence[Item] and not List[Item] if applicable.
I think we both agree, not sure which part of my comment you think is contradictory.
As an argument type, Iterable is permissive (generic).
As a return type, Iterable is restrictive (specific).
This is an odd complaint. typing.Sequence[T] has been there since the first iteration of typing (3.5), for exactly that use case, along with many related collection types.
https://docs.python.org/3/library/typing.html
mypy isn’t perfect, but it’s sure better than making things up without any checks; you’re going to want it for all but the smallest projects.
Dynamically typed code is 1/3rd the size of statically typed code, that means that one developer who is using dynamic typing is equivalent to 3 developers using statically typed code via MyPy.
Since the code is 1/3rd of the size it contains 1/3rd of the bugs.
This is confirmed by all the studies that have been done on the topic.
If you use a static type checking with Python, you have increased your development time by 3 and your bug count by 3.
Static typing's advantage is that the code runs a lot faster but that's only true if the language itself is statically typed. So with Python you have just screwed up.
This is absolutely not true.
> Since the code is 1/3rd of the size it contains 1/3rd of the bugs.
That is made up and contrary to all empirical evidence I've ever collected.
I'd be curious if you have a source, but I doubt it.
Infact, you could just try it out for yourself.
But here is your internet source for this blatantly obvious fact: https://games.greggman.com/game/dynamic-typing-static-typing...
You should use it where it makes sense, and not where it doesn’t. I haven’t used any of Ruby’s type checkers, but Python makes this easy enough; make what has a reason to be dynamic dynamic, and have static safety rails everywhere else.
(This is true with many “statically typed” languages that have dynamic escape hatches, too, not just traditionally “scripting” languages.)
Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.
Depending on the language and the particular project, my sweet spot for test coverage is between 30-70%, testing the tricky bits.
I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
You can have bad unittests which make the system worse and you would be better of without them. You can also have useless unittests with 100% coverage, which is pretty much the same as bad tests because more code means more bugs and more work. Unittests are also code after all.
The only thing you can say about a very low coverage is that you probably don't have good tests. That's not a very useful metric, since you likely already know that.
The metric 'coverage' is almost useless. Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number. I found it is very useful though if you can generate detailed reports on each line of code or better yet, each branch in the code, indicating whether that line or branch is tested. Eyeball all the lines which don't have tests and ask yourself: would it be useful to add a test exercising this codepath? How do I make sure it works and what cases can I think of that could go wrong? This doesn't automatically lead to good tests, but it helps you spot where you should focus your testing efforts.
Code coverage is a good tool to help think of test cases, as a metric for the total codebase it is nearly useless.
When a measure becomes a target, it ceases to be a good measure.
It takes immense discipline to actually let go of a metric to keep it valuable.
It's a red flag to blame high coverage for fragile tests. Use narrow public component interfaces to reach code parts and you simultaneously gain robust tests which can be used during refactoring and you can be guided by coverage to generate test cases. Bob Martin has a great article: https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...
Absolutely not. This leads to testing being invasive and driving the design of your software, usually at the cost of something else (like readability). Testing is a tool, you can't let it turn into a goal.
> Testing is a tool, you can't let it turn into a goal.
Yep, and I use testing as a tool to be sure we ship quality code. It's 2x important for our case, we don't have control on hosts where our product is run and 100% coverage was a salvation. We even start to ship new versions without any manual QA.
If your goal is 100% coverage then it will turn testing into ritual and only give you the illusion of quality. Instead of testing inputs and edge cases, you will focus on testing lines of code.
There's a good illustration of uselessness of 100% coverage in one of Raymond Hettinger talks: https://www.youtube.com/watch?v=ARKbfWk4Xyw
> I use testing as a tool to be sure we ship quality code
I suspect we have different definitions of quality, and your might include testing, so I doubt I will be able to convince you.
> This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.
It's a riff on Boring Technology, see https://boringtechnology.club/
Terrible advice not to use type hints and this reason makes no sense. There's already pretty good support for Sequence and Iterable and so on, and if you run into a place where you really can't write down the types (e.g. kwargs, which a lot of Python programmers abuse), then you can use Any.
Blows my mind how allergic Python programmers are to static typing despite the huge and obvious benefits.
It's true that Python's static typing does suck balls compared to most languages, but they're still a gazillion times better than nothing, and most of the reason they suck so much is that so many Python developers don't use them!
Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before.
All those big changes introduce commits that make git bisect generally slower. Which might be awful if you also have some C code to recompile at every step of bisecting.
Then add black as part of your environment with an specific version...
Reformatting the whole code every version isn't so good. It's also very slow.
Set black up in the pre-commit with a specific version. When you make a commit it will black the files being committed using the specific version of black. As it's a subset, it's fast. As it's a specific version, it's not going back and forth.
I hope this solves your issues.
The further you get away from the project folder the more likely each developer is to have a different environment.
Why? It is expected for the thing to run on different python versions and different setups… what's the point of forcing developers to a uniformity that will not exist?
It's actually better to NOT have this uniformity, so issues can get fixed before the end users complain about them.
Any team of developers who aren't using the exact same environment are going to run into conflicts.
At the very least, there must be a CI job that runs quality gates in a single environment in a PR and refuses to merge until the code is correct. The simplest way is to just fail the build if the job results in modified code, which leaves it to the dev to "get things right". Or you could have the job do the rewriting for simplicity. Just assuming the devs did things the right way before shipping their code is literally problems waiting to happen.
To avoid CI being a bottleneck, the devs should be developing using the same environment as the CI qualify gates (or just running them locally before pushing) with the same environment. The two simple ways to do this are a Docker image or a VM. People who hate that ("kids today and their Docker! get off my lawn!!") could theoretically use pyenv or poetry to install exact versions of all the Python stuff, but different system deps would still lead to problems.
You've never done any open source development I guess?
Do you think all the kernel developers run the same distribution, the same IDE, the same compiler version? LOL.
Same applies for most open source projects.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
In Python, the easiest way to achieve this is using Poetry, which creates a lock file so that all developers are using a consistent set of versions. In other languages, this is generally the default configuration of the standard package manager.
Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes?
> Which might be awful if you also have some C code to recompile at every step of bisecting.
That reminds me, I've got to try out ccache (https://ccache.dev/ ) for my project. My full compile is one minute, but the three files that take longest to compile rarely change.
And testing 1 extra step could only add a 1 hour build more, yes?
This is not isort! isort has never done that. And it has a formatting guarantee across the major versions that it actively tests against projects online that use it on every single commit to the repository: https://pycqa.github.io/isort/docs/major_releases/release_po...
You should never develop using the system Python interpreter. I recommend pyenv [0] to manage the installed interpreters, with a virtual environment for the actual dependencies.
Yes yes… never ever make the software run in a realistic scenario! You might end up finding some bugs and that would be bad! (I'm being sarcastic)
use pre-commit https://pre-commit.com/ so that everyone is on the same version for commits.
Not using a formatter at all is clearly worse than either option.
why?
Do you hate terse diffs in git?
I think the sane part of the software engineering world has realised that auto-formatting is just the right way to do it, and the people that disagree just haven't figured out that they're wrong yet.
Maybe you meant "why is Black specifically better than no autoformatting, given that it isn't perfectly stable across versions?" in which case the answer is:
a) In practice it is very stable. Minor changes are easily worth the benefits.
b) They have a stability guarantee of one calendar year which seems reasonable: https://black.readthedocs.io/en/stable/the_black_code_style/...
c) You can pin the version!!
https://github.com/cjolowicz/cookiecutter-hypermodern-python
I would go so far as to say that the hypermodern template, nomenclature aside, is strictly better than the recommendations that the OP put forward both here and in the previous essay on dependency management. Poetry and ruff, for instance, are both very good tools — and I can understand _not_ recommending them for one reason or another but to not even mention them strikes me as worrisome.
My concern is a) It needs to be reliable (don't wanna spend a ton of time chasing bugs later on) b) How can I write the actual code better? I see what pro devs write and they use smarter language features or better organization of the code itself that makes it faster and reliable, I wish I could learn that explicitly somewhere.
I mean, just the 2.7->3.0 jump was big for me because since I don't code regularly that meant googling errors a lot basically. Even now, I dread new python versions because some dependency would start using those features and that means I have to use venv to get that small script to work and then figure out how to troubleshoot bugs in that other lib's code with the new feature so I can do a PR for them.
I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly. Those are just not suitable for people whose day job is not coding and migrating to new versions, supporting code bases, messing with build systems, unit tests, qa,ci,etc... coding is a tool for me, not the centerpiece of all I do. But python is still great despite all that.
What do you mean by "drastic" features "quickly"? Python releases new version once a year these days, and upgrading our Django-based source code with 150 dependencies from 3.4 to 3.11 literally meant switching out the python version in our CI configuration and README.rst every once in a while, no code changes were necessary for any of those jumps...
Our developer README also contains a guide how to set-up and use pyenv and it's virtualenv plugin which makes installing new python versions and managing virtualenvs easy, just pyenv install, pyenv virtualenv, pyenv local, and your shell automatically uses the correct virtualenv whenever you're anywhere inside your project folder...
jumping to python3 was big, but you had plenty of time to prepare for that and plenty of good utilities to make the jump easier (2to3, six, ...). python2.7 itself was released 18 months after python3.0, and by the time python2.7's support ended, python3.8 was already out...
Second, yes, all you have to do is switch out the python version to upgrade but let's say you start using f-strings that means all of your users (doesn't apply to django since it is server software) have to upgrade to the right python version including all the deps. But what if your project is a library? That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
New versions should come out no more often than every 3-4 years imho and even then every effort should be made to have those features backward compatible like have a tool that will degrade scripts to be usable on a previous language version.
2.7 was supported for 10 years and it's support ended 2 years ago. There's been ample time to upgrade the code or look for an alternative. If I "absolutely needed" to use a piece of code that I didn't write, is for an unsupported platform and is itself unsupported, I'd absolutely find the time for it. As a developer if I use a library that hasn't been touched for 3 years it's a red flag and I start to look for alternative libraries or forking the code.
> That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
if your distro doesn't support the latest python version you're probably on a very old distro. For example python3.11 installs fine on all supported versions of Ubuntu (18.04+) and Debian (10+) and both Windows (8.1+) and macOS (10.15+). And python3.9 installs fine even on centos7 (released in 2014) and still supports the vast majority of python libraries.
If you're on an OS nearing or past its end of support, you can't reasonably expect all the latest software to work on it. And it's usually fine to just use an older version of python / libraries until you're ready to update.
> New versions should come out no more often than every 3-4 years imho
If new versions came out every 3-4 years, that would mean they would have more drastic changes, because the smaller changes would accumulate over that duration. The longer the "new features" are out, the longer users have to upgrade their system and the longer developers can take getting used to them.
But in the end, it doesn't really matter how often a new version comes out but rather how long the old versions should be supported, right? And I think it's up to the library authors to decide how long to support older versions, not the authors of the programming language.
If a dependency breaks compatibility with earlier Python versions because the author wants to use a fancy new feature is not really the fault of Python, is it? Library authors should target the earliest supported Python version they can.
Being backwards compatible (at which Python has been doing a good job since the 2->3 fiasco) is one thing, but trying to be forwards compatible is something else.
Are you suggesting that Python developers should only ship bug fixes so that Python 3.0 can still run code written for Python 3.11?
In 3.8 someone decided that they didn't like the way people were excepting the Exception for cancelled asycnio tasks. So they changed the cancelled task exception to inherit from base exception instead of exception. This meant a bunch of well used libraries immediately had a load of subtle bugs that in normal operation just didn't happen. I can't remember the exact details but I think when the bug did happen the task queue would just continue to grow until we ran out of memory.
This change wasn't a bug fix, more an optimization or an attempt to get people to code a certain way.
I'm all in favour of bug fixes, but Devs shouldn't have to worry about minor upgrades breaking everything.
I have a library… most downloaded version is 3 years old. The newer versions are massively faster but nobody uses them.
But it’s totally reasonable to pin the private requirements that you develop it against (listed in requirements.txt, poetry.lock, or similar), updating them every so often during the course of development, so that contributors can use a consistent set of tools.
About typings: I agree the eco-system is not mature enough, especially for some frameworks such as Django, but the effort is still valuable and in many cases the static analysis provided by mypy is more useful than not using it at all. So I would suggest to try do your best to make it work.
When python converges on consistent typing across its extended numpy and pandas ecosystem, I believe we will be able to move towards a fully JIT'd language.
Unless they actually go ahead with the deferred evaluation of types (PEP 563), make all types strings at runtime and make it impossible to know which type they actually are. :)
But they will probably not: https://discuss.python.org/t/type-annotations-pep-649-and-pe...
But it could be a breaking change in the language. As it is, I can run this "a: str = 3" and it will work.
On Ubuntu and Windows I use Poetry [0], and it works, although it has (had?) some quirks during the installation on Windows. I liked its portability and lockfile format though.
A few years ago I used conda [1], which was nice because it came batteries included especially for Deep Learning stuff. I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi.
And then there are also Docker Images, which I use if I want to give an application to somebody that "just works".
What's your method of choice?
Agreed. I like docker images for smallish portable scripts. At home I can develop on my Mac and port it to a Raspberry PI or another x86 Windows/Linux box.
Planning on running a docker swarm with a few Pi’s to see how it works.