Rachel says that a thread is only ever doing one thing at a time - it is handling one request, not many. But that's only true when you do CPU bound work. There is no way to write code using blocking IO-style code without using some form of event loop (gevent, async/await). You cannot spin up 100K native threads to handle 100K requests that are IO bound (which is very common in a microservice architecture, since requests will very quickly block on requests to other services). Or well, you can, but the native thread context switch overhead is very quickly going to grind the machine to a halt as you grow.
I'm a big fan of gevent, and while it does have these shortcomings - they are there because it's all on top of Python, a language which started out with the classic async model (native threads), rather than this model.
Golang, on the other hand, doesn't suffer from them as it was designed from the get-go with this threading model in mind. So it allows you to write blocking style code and get the benefits of an event loop (you never have to think about whether you need to await this operation). And on the other hand, goroutines can be preempted if they spend too long doing CPU work, just like normal threads.
Why? Bigger MySQL servers regularly use 10k native pthreads which spend most of their time waiting on IO.
Context switch overhead isn’t linear with the number of threads! The Linux scheduler runs O(logN). Moving from 10k MySQL threads to 1 thread per core will give you less than 10% extra throughput.
Doing IO means context switching. Goroutines don’t get a free pass here.
For example, the cost of the context switch itself (storing all registers) is more significant with native threads.
Just try spinning up 100K threads that each print a line and then sleep for 10ms, and see how high your CPU usage gets.
Also, doing IO does not necessarily mean context switching - it means calling into the kernel (system calls). If you use an async IO operation (read/write from a socket) and then continue to the next thread, by the time you're done with all ready threads, you're likely to have some sockets ready to read from again, so you might not context switch at all. Kernel developers are working on even reducing the need for syscalls with io_uring, which is designed to allow you to perform IO without system calls.
The Varnish server does that all the time and it is not a problem.
Modern OS handle a large number of threads much better than 10 years ago.
In case of python however, you can not do that due to the GIL and the fact python handle native threading terribly.
> We rarely recommend running with more than 5000 threads. If you seem to need more than 5000 threads, it’s very likely that there is something not quite right about your setup, and you should investigate elsewhere before you increase the maximum value.
See the conversation here: https://news.ycombinator.com/item?id=22515100
You should just update your reasoning and say you can’t have 100M threads waiting for blocking IO.
At a previous job, I wrote a wrapper that would compare the last execution time of the event loop scheduler, and if it was larger than some value, it would yield (by sleeping for 0ms). This was intended for use in loops with varying length, which might be long and could be short depending on some external factors, but yielding on each loop iteration made them slower by an order of magnitude.
IIRC It was sending messages to RabbitMQ, which somewhere down the line is writing to a socket, but not necessarily blocking - if the socket manages to flush its buffer fast enough (faster than the processing code can send messages), writing to it may never block, resulting in a CPU bound loop (since the loop may perform work other than sending).
We didn't want people to have to think too hard about how their loop was going to behave (especially as it might mean reasoning about the internals of a third party library), and so the wrapper was born. If your loop was short enough, all it did was compare ints so the cost was negligible.
Isn't node.js IO native threaded?
This is actually an important property of JavaScript that allows it to work without locks (as long as you do not await, your code runs completely synchronously and will not be interrupted). It's not a flaw, but definitely requires awareness.
Gevent code that doesn't spawn more than one native thread (the one that runs the event loop) also has this property - you don't need locks as long as you do not perform IO. In Python's case it can be more tricky, as you might end up yielding to the event loop by indirectly performing IO when logging or something similar.
In JavaScript, the only case, AFAIK, where this can happen is when you await. Nothing else will cause you to yield to the event loop.
From a high level you can think of it as a single thread that can only context switch during an IO call. Otherwise the function has to run and block all the way to termination. <---- This is what rachel is complaining about.
I get that once you reach full utelization, the best the kernel can do is to make it fair, and that's going to cost. That's not a general statement like "100k threads grinds the machine to a halt".
You can just pass `--preload` to have gunicorn load the application once. If you're using a standard framework like Django or Flask and not doing anything obviously insane then this works really well and without much effort. Yeah I'm sure some dumb libraries do some dumb things, but that's on them, and you for using those libraries. Same as any language.
If you want to stick your nose up at Python and state outright "I will not write a service in it" then that's up to you, it just comes across as your loss rather than a damning condemnation of the language and it's ecosystem from Rachel By The Bay, an all-knowing and experienced higher power. I guess everyone else will keep quickly shipping value to customers with it while you worry about five processes waking up from a system call at once or an extra 150mb of memory usage.
Every time you access an object, its reference count is mutated, which will the memory page to be copied.
There are workarounds if you're willing to mess with the Python interpreter: https://instagram-engineering.com/copy-on-write-friendly-pyt...
isn't it CPython specific though, and may not be observed when running on PyPy?
I dug into it ~year ago, Django loads almost everything lazily, so simple --preload did next to nothing. I had to write code to load app for real at import time, exact thing article and common wisdom tells us not to do.
I believe the i18n system is also lazily loaded and depending on the languages you configure it can take up a fair bit of memory.
That is not true. RotB is describing saftware inefficiencies that we learnt to do without 20 years plus ago.
Because the Python hackers that built all these tools did not pay attention, when they built new tools they recreated the old problems.
Worik's 23.6918th rule of creativity: It is easier to write than read.
It's not always trivial to ensure none of your dependencies have import-time side effects. Sometimes the productivity/business benefit provided by the depedendency outweighs the pain introduced by the side effects.
With the current state of ecosystems, this quality-vs-quantity mutual exclusivity is much less pronounced. These days, you can fire up these services as quick or quicker than in Python with better performance and resource usage that is also more maintainable. Unless you speak of highly ecosystem-dependant libraries (e.g. ML), Python defenses that rely on time to market say more about the author's narrow comfort than general expediency.
If all you're doing is basic web stuff then sure you can do it in Lang du jour but even after a decade go doesn't have a competent xml library for example.
Time to market is killer. Python buys you time to determine whether your product is even worth building. Deployment sucks though.
e: I literally could not have written the services in my current role in a language other than Java or Python without replicating 100kloc libraries. Java would have required a bunch of work to integrate with the other services we had. So: python. If that costs me an extra $1k/mo for servers but gets us a customer paying $100k a month, was it wasteful?
Narrowing in on the rather specific point about shipping things with Python and ignoring the larger argument that it doesn't freaking matter if things are not as efficient as they could possibly be is quite odd to be honest. I'm sure some of the arguments in the blog post would apply to whatever language you had in mind while writing your reply.
If I were her, I’d put that on my CV. :D
The author seems to be saying something about how if a worker is busy doing CPU intensive work (is decoding JSON really that intensive?) then other requests accepted by that worker have to wait for that work to complete before they can respond, and the client might timeout while waiting?
If that's the case:
1. Wouldn't this affect any language/framework that uses a cooperative concurrency model, including node.js and ASP.NET or even Python's async/await based frameworks? How is this problem specific to Python/Gunicorn/Gevent?
2. What would be a better alternative? The author says something about using actual OS-level threads but I thought the whole point of green threads was that they are cheaper than thread switching?
2. Green threads have lower overhead, but it's a false economy if it causes you to do needlessly redo work because of timeouts that could have been avoided.
Which it seems it must because the kernel doesn't have the insight to know whether a green thread is doing that epoll because it's ACTUALLY idle or because it's not but it's willing to try to juggle a second (or third) thing while it has something on the back burner. So the kernel indiscriminately assigns work to threads without regard for whether they are juggling a lot or nothing.
Whereas with native threads, they never ask the kernel for more work while they're blocked on something else because they are literally blocked and thus won't be making that epoll system call.
(The article also mentions something about LIFO policy, which exacerbates the problem because it favors assigning work to the process which is likely to already have most of it.)
Also, isn't the problem that JSON decoding (or whatever computation) simply block the thread and the other green threads cannot proceed at all, because there are simply no safepoints (yield points) inside these low level functions?
And in all these cases shouldn't the application estimate work (eg. in case of JSON if the string is longer than 100K), and if it's too big just put it on a dedicated heavy compute N:N thread pool?
For Python it's best practice anyway because the GIL, no?
https://docs.python.org/3/library/asyncio-eventloop.html#asy...
Node.js will have that issue and in fact the stdlib JSON encoding/decoding can't even be paused so once you start a processing something you're stuck until it's done. You could, however, write an incremental serializer/deserializer that could spread processing out across many event loop cycles to mitigate.
Go, ASP.NET, and others not so much, depending, because the schedulers can pause and resume the tasks(on top of being threaded).
I think she's against anything that has that problem. Not every green thread implementation has that problem. For example Go doesn't have that problem. Because there were 4 CPU threads (I think) and only 2 things needing to be done, with Go's M:N scheduling those 2 things would be sure to both be running.
Yes. Decoding JSON with Python is CPU-intensive.
This a very simple shell script around Python, that is designed from the get-go to crash with an exception. However, it may not be the exception you expect:
n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')"
left="$(yes [ | head -n "$n" | tr -d '\n')"
echo "$left" | python3 -c 'import json; print(json.loads(input()))'
You won't. You'll hit RecursionError.
Because, despite how badly Python deals with recursion, the JSON library depends on it extensively. Which means that there is a huge stack-tree being built every time you try and decode JSON in Python (Dictionaries, function call overhead, etc, etc.), only for it to be thrown away.
in Python, everything is generally CPU intensive compared to what it would be in compiled languages, even though things like JSON decoding are usually happening in a C library, Python programs that do close to nothing still use way more CPU than you would if you were running in the JVM, or Go, C, whatever.
> Wouldn't this affect any language/framework that uses a cooperative concurrency model, including node.js and ASP.NET or even Python's async/await based frameworks? How is this problem specific to Python/Gunicorn/Gevent?
CPU bound-ness affects all of these platforms, yes. It affects Python and other intepreted languages the most however because these platforms get the most CPU-bound the most quickly. Also, applications that are written in scripting languages tend to have a lot of business logic going on in the first place; after all, if you just wanted to serve static pages you could use Apache with the Event NPM and if you wanted to proxy HTTP requests you'd use HAProxy; both event-based systems that are very much not CPU bound.
But yes, most importantly, Python's asyncio system is completely impacted by these same issues and I would have preferred she address that, as asyncio is part of the standard library now and is way more popular than gevent.
> What would be a better alternative? The author says something about using actual OS-level threads but I thought the whole point of green threads was that they are cheaper than thread switching?
I will grant she lost me a bit with the "use a real RPC system with <feature> <feature> <feature>" thing, and additionally the "load the application in the child process" thing is pretty typical, a worker process should obviously have either threads or greenthreads in use so that each process can handle multiple concurrent requests, but only as many as you'd want handled effectively by one core since the GIL is going to enforce that (another thing you wouldn't have to deal with in other languages such as the above mentioned compiled languages), but it's typical that child processes are going to have a mostly original copy of things.
But as far as the "context switching" thing, I've yet to see benchmarks that show the overhead of OS-level context switching actually being more of a performance burden than the less frequent, but more work intensive context switching that user-space schemes like asyncio have to use. If you are writing a logic-heavy, or even a logic-just-a-bit service that receives Python requests you will also have to worry about CPU-bound issues all the time. Using regular threads with processes, like what you get using something like mod_wsgi, will allow individual processes to attend to web requests more evenly. With mod_wsgi you can configure worker daemons that run multiple OS level threads and you can also have multiple daemon processes.
I'm not sure if the multi-process model used by mod_wsgi has solved the accept() problem, however in my experience the bigger problem is when a service configures itself to allow for 1000 greenlets within each process, while each process is realistically capable from a CPU perspective of handling maybe 5 or 10 concurrent requests, there's no mechanism that ensures that each process gets an even balance of requests. That is, you might have all your requests waiting in one process, because you told them it can process 1000 at a time, while other processes are idle.
TL;DR I'm in the "event based programming is extremely overrated in Python" camp.
Yeah but that's not exactly shocking news to anyone, is it? People generally choose Python for other reasons (productivity, library ecosystem, etc.) because the performance is "good enough" for most web apps, and if you reach a traffic level where performance becomes an issue that's a good problem that you can optimize for later. (Like Facebook did with PHP, Twitter did with Ruby, etc.)
> But yes, most importantly, Python's asyncio system is completely impacted by these same issues and I would have preferred she address that, as asyncio is part of the standard library now and is way more popular than gevent.
Right, the blog post gave me the impression she was calling out the combination of Python/Gunicorn/Gevent specifically for some reason. But if the underlying goal was to just point out that Python is slow then I am curious what people think the right solution is? Just switch out of Python and use Go or something else?
On the first point, yeah Rachel’s posts are kinda snarky sometimes, but some of us find that entertaining particularly when they are highly detailed and thoroughly researched. I’ve worked with Rachel and she’s among the best “deep-dive” userspace-to-network driver problem solvers around. She knows her shit and we’re lucky she takes the time to put hard-earned lessons on the net for others to benefit from.
As for “microservices written in Python trading a bunch of sloppy JSON around via HTTP” is bad engineering: it is bad engineering, sometimes the flavor of the month is rancid (CORBA, multiple implementation inheritance, XSLT, I could go on). Introducing network boundaries where function calls would work is a bad idea, as anyone who’s dealt seriously with distributed systems for a living knows. JSON-over-HTTP for RPC is lazy, inefficient in machine time and engineering effort, and trivially obsolete in a world where Protocol Buffers/gRPC or Thrift and their ilk are so mature.
Now none of this is to say you should rewrite your system if it’s built that way, legacy stuff is a thing. But Rachel wrote a detailed piece on why you are asking for trouble if you build new stuff like this and people are, in my humble opinion, shooting the messenger.
Laziness is a virtue in our profession :) huge "citation required" on it being more efficient in engineering effort and most importantly it's ubiquitous and interoperable with any language without having to rely on grpc or thrift implementation and tools for that language/platform.
Re machine time efficiency - to the extent I understood what she was actually complaining about simply none of those issues are attributable to json-over-http being used.
And the calculus of cost doesn't even always work out the same.
I genuinely don't see much of a lesson to learn from this particular blogpost, and it appears neither did many others in HN. If there is one, beyond "don't use x", it's hard to find it.
I get the impression that this particular post is being upvoted to the top of HN because of who the author is, not necessarily because this post itself has value. This results in a whole bunch of others reading it, wondering why they're wasting their time with such a rambling post.
1. Load before forking.
2. Remember that green threads tend to have problems with fairness of scheduling.
3. JSON decoding gobbles CPU.
4. Scheduling fairness problems increase response time variance.
4½. Green threads also increase it.
5. Don't forget about retries of timed-out requests into account in protocol design; idempotence is the simplest solution when you can use it.
6. Wake-one semantics to avoid the thundering herd are important for performance when you have multiple threads, and Gunicorn has that thundering herd problem, so you probably don't want to be running it this way on a 64-core box with hyperthreading. (The problem is of course less severe than it was for Apache because the green threads don't thunder.)
7. Gevent uses epoll, not select, poll, or RT signals
8. EAGAIN and SIGPIPE if you didn't know about those. (Somebody is in today's lucky ten thousand.)
9. What kinds of mechanisms “tend to show up given time in a battle-tested [network server] system.”
10. Your systems don't have to be fragile pieces of shit.
Also, I feel like she (intentionally maybe) is conflating concurrency and parallelism. These "green thread" systems provide concurrency, but not parallelism. That should be something people are aware of when they use them.
[1] https://capnproto.org/faq.html#how-do-i-make-a-field-require...
Hmm, you seem to be citing me. To clarify:
1. I didn't design Protobuf. I just rewrote the implementation (created version 2) and open sourced it.
2. I don't think it's awful. In fact, I think it's best-of-breed for what it is, and certainly a much better choice than JSON-over-HTTP. Yes, in Cap'n Proto I've added a bunch of features which Protobuf doesn't have, like zero-copy and promise pipelining, and I obviously think these features make it better tech. But, to be fair, these ideas make Cap'n Proto more complicated, and whether that complexity is worth it is still very much unproven at the scale that Google uses Protobuf and gRPC/Stubby.
> These "green thread" systems provide concurrency, but not parallelism.
I'm not sure if these specific definitions of "concurrency" and "parallelism" are universal. I wasn't aware of them, at least.
As the maintainer of about 5 little services with this structure I have vowed never to write another one. The memory overhead alone is a source of eternal irritation ("Surely there must be a better way....").
Echoing other commenters here, the real cost isn't actually discussed. Namely that there is a solution to some of these problems (re long running tasks?), but it carries with it a major increase in complexity. Its name is Celery and oh boy have fun with the ops overhead that that is going to induce.
A while back I did some unscientific benchmarking of the various worker classes for python3.6 and pypy3 (7.0 at the time I think?). Quoting my summary notes: 1. "pypy3 with sync worker has roughly the same performance, gevent is monstrously slow gthread is about 20 rps slower than sync (1s over 1k requests), sync can get up to ~150rps" 2. "pypy3 clearly faster with tornado than anything running 3.6" 3. "pypy3 is also about 4x faster when dumping nt straight from the database, peaking at about 80MBps to disk on the same computer while python3.6 hits ~20MBps"
I won't mention the workload because it was the same for both implementations and would only confuse the point, which is that there are better solutions out there in python land if you are stuck with one of these systems.
One thing I would love to hear from others is how other runtimes do this in a sane and performant way. What is the better solution left implicit in this post?
I'm surprised how often devs treat this distinction as architecturally meaningful. Web requests are just RPCs with some of the parameters standardized and multiple surfaces for parameters and return values - query string, headers, body. This is completely orthogonal to the strategy used to schedule IO, concurrency, etc.
These systems tend to take care of a bunch of easy to mess up logic which tends to be accumulate around any system that wants to send something and not mess up. Any sufficiently old web request system tends to look identical to a high level RPC system designed badly.
So choosing an RPC system should give you all the features you’d eventually end up building around using web requests without spending your time rewriting it.
But a dedicatd RPC stack such as grpc or thrift constraints my architecture in other ways, that json-over-http doesn't.
And what are those unavoidable costs of REST that are not just the wire overhead of HTTP ?
Also, TLS is not really ideally lightweight for RPCs, but you should absolutely encrypt your RPC traffic (imo.) So I really think the whole stack is out.
(P.S.: If you are wondering what kinds of 'lightweight' replacements for TLS exist, I think my personal favorite attempt is CurveCP, although it is a bit dated nowadays. I wouldn't often recommend people roll their own, but you could certainly do something simple with NaCl/libsodium directly. Maybe QUIC also fits the bill?)
No, it doesn't. But there is no evidence this overhead actually mattered here. It usually doesn't because the CPUs easily outrun whatever bandwidth is available which is why JSON over HTTP is fine 99% of the time. There is absolutely nothing in this blog post that shows that's not also the case here. No rationale is provided as to how a strongly typed RPC mechanism would solve any actual problems the services is having.
So we're left with guesswork and the authors hang-ups about HTTP vs some as yet unnamed RPC solution.
Also, Gunicorn? Thundering herd? These are solved problems. Space the toy proxy and use something real like haproxy. At a minimum.
Finally, none of this griping about HTTP vs RPC actually addresses the _actual_ problem: the server can't process requests in a timely manner. That points to some deeper inefficiency or design issue that likely has nothing to do with Python or Gunicorn or Gevent at all. We're not given any insight as to what the hell the server is doing with all that CPU. Or why the client isn't using a protocol intended for long running processes; RPC schemes have timeouts too ya know....
This reads like a poorly informed rant.
Because of the GIL, you can't make predictions at the same time you're processing network IO, which means that you need multiple processes to respond to clients quickly and keep the CPU busy. But models use a lot of memory and so you can't run all THAT many processes.
I actually did get the load-then-fork, copy-on-write thing to work, but Python's garbage collections cause things to get moved around in memory and triggers copying and makes the processes gradually consume more and more memory as the model becomes less and less shared. Ok, so then you can terminate and re-fork the processes periodically, and avoid OOM errors, but there's still a lot of memory overhead and CPU usage is pretty low even when there are lots of clients waiting and...
You know I hear Julia is pretty mature these days and hey didn't Google release this nifty C++ library for ML and notebooks aren't THAT much easier. Between the GIL and the complete insanity that is python packaging, I think it's actually the worst possible language to use for ML.
She is basically complaining about a pattern that was popularized by NodeJS and emulated in python by older libraries like gevent, twisted and tornado. Currently python3 uses keywords async/await as an API around the same concepts implemented in the older libraries.
This has nothing to do with GIL.
Why not? If the model is a Python wrapper around some C/C++ library, then GIL can be released and this is actually a recommended approach used by almost any CPU-intensive python libraries - https://docs.python.org/3/c-api/init.html#releasing-the-gil-... You can have parallel computations inside your wrapped C extension, while Python interpreter is processing IO.
> So how do you keep this kind of monster running? First, you make sure you never allow it to use too much of the CPU, because empirically, it'll mean that you're getting distracted too much and are timing out some requests while chasing down others. You set your system to "elastically scale up" at some pitiful utilization level, like 25-30% of the entire machine.
Letting a Python web service, written in your framework of choice, perform CPU-bound work is just bad design. A Python web service should essentially be router for data, controlling authentication/authorization, I/O formatting, and not much else. CPU intensive tasks should be submitted to a worker queue and handled out of process. Since this is Python we don't have the luxury of using threads to perform CPU-bound work (because of the Global Interpreter Lock).
You can have it with threads if the CPU-bound work is done inside a C extension - https://docs.python.org/3/c-api/init.html#releasing-the-gil-...
The question, however, is why one would use gevent at this point in Python's evolution. There's async await now, and things like FastAPI. If you want to use, say, the Django ecosystem, use Nginx and uWSGI and be done with it. Maybe you need to spend some more resources to deploy your Python. Okay. Is that a problem? Why are you using Python? Is it because it's quick to use and helps you solve problems faster with its gigantic, mature ecosystem that lets you focus on your business logic? Then this, while admittedly not great, is going to be a rounding error. Is it because you began using it in the aforementioned case and now you're boxed into an expensive corner and you need to figure out how to scale parts of your presumably useful production architecture serving a Very Useful Application?
Maybe you need to start splitting up your architecture into separate services, so that you can use Python for the things that it does well and use some other technology for the parts that aren't I/O bound and could benefit from that. But that's not this article is about. This article is about someone making the wrong choices when better choices existed and then making a categorical decision against using Python for a service. I'd say that's what "we have to talk about" if you ask me.
The problems alluded to are, yes, solvable in python. But they also seem endemic in python systems.
When everyone who uses the tool uses it wrong, maybe it's not the user's fault.
(That said, I generally do think there's a time and place for python systems or web apps. That time is generally when speed and maintainability is significantly less important than flexibility)
Yes, though that doesn't mean it is necessarily the code's fault.
Honestly, I was very confused by this article, because I thought everyone understood what was going on, the trade-offs involved, and how that ought to impact your design decisions.
It's not that Gevent'd Gunicorn is intrinsically a bad thing. You're going for cooperative multi-tasking/concurrency, so no preemptive multi-tasking support. This creates potential challenges with fair scheduling if you have real-time constraints like timeouts... so you design accordingly.
One of the advantages of this model is you do indeed need less memory (and often a little less CPU) to handle high load levels. It's not like you are intrinsically better off if you use Python in a forking model. You can still end up so CPU bound that you timeout handling requests... the only difference is you'll get fairer splitting of the CPU's time across tasks. It can actually get worse if you get lost in an infinite series of context switches (yes, there are ways to mitigate this problem... although they can create fair scheduling problems... it's a natural tension), or worse still, start swapping.
If the notion that running out of CPU might mean you have timeouts hasn't occurred to you...
I'm not the GP, but I guess that a tool that is
> quick to use and helps you solve problems faster with its gigantic, mature ecosystem that lets you focus on your business logic
Can never cover all bases perfectly, and is generally great when starting out, but ultimately not built to be very forgiving when grown too much.
> now you're boxed into an expensive corner and you need to figure out how to scale
When you get to this point, and the requirements start to be more focused on performance, then it's time to start switching Python out. That does not devaule Python in the earlier stages of development and operation.
The point being that Python is the right tool for getting stuff working quick, not to getting stuff executing quick.
gevent and gunicorn were good attempts to remedy a bad situation. async/await is the solution that the Python community is coalescing around. Even with Django, there are active efforts to support ASGI. [1]
[1]: https://docs.djangoproject.com/en/3.0/howto/deployment/asgi/...
In high-level languages with virtual machines and/or garbage collectors, the runtime system should be solely responsible for scheduling green threads around IO entry points, all without special syntactic markers. GHC has it right (https://www.aosabook.org/en/posa/warp.html), Gevent was a right development with on-par async performance metrics (https://gist.github.com/rfyiamcool/41d4004b7fd46516d0b4f34f6...), that had a standard synchronous coding style. It could be adopted into the core language and improved further without splitting the community.
I wasn't aware of this particular inefficiency, but gevent is still fulfilling its purpose for me very well, and I see no reason to change. I like lightweight threads and thinking in terms of background jobs and dividing up work instead of remembering what things to annotate and when. I use locks if I need predictability. I like Python because I can develop quickly with it, and I can do so even faster with gevent while still getting more than enough performance.
This is not trivial to set up correctly and other ecosystems already have sane, scalable, secure-out-of-the-box deployment schemes figured out.
It's an amazing library and a very unique way to write cooperatively-scheduled applications. Best of all it works with existing libraries and doesn't require special "asyncio" implementations from top to bottom. It's not a silver bullet, but don't fool yourself that asyncio is because it's been blessed.
What you call "special" asyncio implementations others would merely call obviously explicit code. Async/await is a powerful syntactic construct. I would never go back to gevent hell after using it.
Let's say your server has 4 CPUs. The conservative option is to limit yourself to 4 requests at a time. But for most web applications, requests use tiny bursts of CPU in between longer spans of I/O, so your CPUs will be mostly idle.
Let's say we want to make better use of our CPUs and accept 40 requests at a time. Some environments (Java, Go, etc) allow any of the 40 requests to run on any of the CPUs. A request will have to wait only if 4+ of the 40 requests currently need to do CPU work.
Some environments (Node, Python, Ruby) allow a process to only use a single CPU at a time (roughly). You could run 40 processes, but that uses a lot of memory. The standard alternative is to do process-per-CPU; for this example we might run 4 processes and give each process 10 concurrent requests.
But now requests will have to wait if more than 1 of the 10 requests in its process needs to do CPU work. This has a higher probability of happening than "4+ out of 40". That's why this setup will result in higher latency.
And there's a bunch more to it. For example, it's slightly more expensive (for cache/NUMA reasons) for a request to switch from one CPU to another, so some high-performance frameworks intentionally pin requests to CPUs, e.g. Nginx, Seastar. A "work-stealing" scheduler tries to strike a balance: requests are pinned to CPUs, but if a CPU is idle it can "steal" a request from another CPU.
The starvation/timeout problem described in the post is strictly more likely to happen in process-per-CPU, sure. But for a ton of web app workloads, the odds of it happening are low, and there are things you can do to improve the situation.
The post also talks about Gunicorn accepting connections inefficiently and that should probably be fixed, but that space has very similar tradeoffs <https://blog.cloudflare.com/the-sad-state-of-linux-socket-ba....
Any of the above (or Sanic) can do ~3K RPS on a single core on a Raspberry Pi (which is where I test things for portability, optimisation and a little fun), and the RAM overhead is generally not that bad small (just did a little "hello world" uvicorn/blacksheep app and I see 22MB resident/10MB shared per worker, and one of my Clojure servers taking up over four times that...)
Those below who complain about the complaints are missing the point.
We (computer programmers as a general class) have not learnt from history. We keep reinventing wheels and each time they are heavier and clunkier.
What we used to do in 40K of scripts now takes two gigabytes in python/django/whateverthehellelse. E.g. mail list servers. Mailman3 hang your head in shame!
> "Why in the hell would you fork then load, instead of load then fork?"
In python it often seems to make little difference. The continual refcount incrementing and decrementing sooner or later touches most everything and causes the copy to happen whether you're mutating an object or not.
I've had some broad thoughts about how one would give cpython the ability to "turn off" gc and refcounting for some "forever" objects which you know you're never going to want to free, but it wouldn't be pretty as it would require segregating these objects into their own arenas to prevent neighbour writes dirtying the whole page anyway...
> A connection arrives on the socket. Linux runs a pass down the list of listeners doing the epoll thing -- all of them! -- and tells every single one of them that something's waiting out there. They each wake up, one after another, a few nanoseconds apart.
Linux is known to have poor fairness with multiple processes listening to the same socket. For most setups that require forking a process, you run a local loadbalancer on box, whether it's haproxy or something else, and have each process listen on its own port. This not only allows you to ensure fairness by whatever load balance policy you want, but also lets you have healthchecks, queueing, etc.
>Meanwhile, that original request is getting old. The request it made has since received a response, but since there's not been an opportunity to flip back to it, the new request is still cooking. Eventually, that new request's computations are done, and it sends back a reply: 200 HTTP/1.1 OK, blah blah blah.
This can happen whether it's an os threaded design or a userspace green-thread runtime. If a process is overloaded, clients can and will timeout on the request. The main difference is in a green-thread runtime it's about overloading the process vs. utilizing all threads. Can make this better by using a local load balancer on box and spreading load evenly. It's also best practice to minimize "blocking" in the application that causes these pauses to happen.
>That's why they fork-then-load. That's why it takes up so much memory, and that's why you can't just have a bunch of these stupid things hanging around, each handling one request at a time and not pulling a "SHINYTHING!" and ignoring one just because another came in. There's just not enough RAM on the machine to let you do this. So, num_cpus + 1 it is.
Delayed imports (because of cyclical dependencies) is bad practice. That being said, forking N processes is standard for languages/runtimes that can only utilize a single core (python, ruby, javascript, etc.).
This is not to say that this solution is ideal -- just that with a small bit of work you can improve the scalability/reliability/behavior under load of these systems by quite a bit.
We have money, let’s just blow it. /s
I noticed in the logs that I am getting a lot of Critical Worker Timeouts and I am wondering if this has anything to do with it.
--Dockerfile snippet--
# Cloud Run concurrency is assumed to be set to 10 but we don't assume that is exact
# See 'https://github.com/benoitc/gunicorn/issues/1801' so disabling concurrency
ENV GUNICORN_CMD_ARGS="-c gunicorn_config.py --workers 1 --threads 1 --timeout 120 --preload"
CMD [ "gunicorn", "pkg.http:app" ]
# Or just use Flask directly if concurrency is set to 1
#CMD [ "python", "cmd/server/main.py" ]
Is the message that epoll and co are lot efficient enough? That’s also true. Api Problems and thundering here are known. And not only limited to Python applications as users. io completion based models (eg throuh uring) solve some of the issues.
Or is this mainly about Python and/or Gevent? If yes, then I don’t understand it, since the described issues can be found in the same way in libuv, node.js, Rust, Netty, etc
Definitely, I am smarter than the guy who wrote this because then I wouldn't have these problems(Or He is smarter and I just didn't ask him about his rationale).
What I design wouldn't run into these BS problems that I have to fix, It just wouldn't run into problems generally. (Or It would have more problems than this one)
I had these conversations with myself at least a thousand times, and then it was just the case in the parentheses.
This is not the whole story of the value of those languages. As the article even alludes to, at small loads or with lots of care this can be made to "work". But it is something that an engineer should know about them before picking them up and using a tool for something it isn't really good at.
[1]: I add this caveat because I don't think there's anything about dynamic scripting languages that makes them intrinsically difficult to thread any moreso than any other category of language, it's just that by an accident of history, they all come to us from the 1990s personal computer world, and they all spent at least a decade cooking and setting and building libraries and communities and developer skillsets before a serious need for threading was even on the horizon.
It's intrinsically single-threaded, yes. But each instance is quite small and they stay out of each others way. Add coroutines and there's a lot you can safely do with Lua that's a real pain to accomplish with Python.
I think there might be some intrinsic factors:
1. Languages like Python don't want to expose the program to undefined behavior. Defining some trivial class, and then manipulating it from two threads at the same time, is not supposed to crash the program or introduce horrible security issues.
2. Languages like Python have "property bag" objects. That means (someone who knows better should check me on this) that most writes in a typical program are hash table operations, rather than primitive stores of an int or whatever. Locking each table separately, or using a fully atomic implementation, can be a significant slowdown in single-threaded programs, compared to using a GIL.
You don't necessarily need one; it depends on what kind of application you have and what its bottleneck is. If your application is network I/O bound, event-driven asynchronous I/O, which is what this article is describing, works fine, and a single process/thread even in a dynamic language like Python can handle a large volume of requests.
The specific issue this article is describing is due to a particular poor implementation of event-driven asynchronous I/O, not a general problem with the entire concept.
If your application is CPU bound, then yes, you need to use threads (or multiple processes), and you shouldn't be trying to mix event-driven asynchronous I/O with that.
Now, using kernel threads uncarefully will lead to different problems, which are famously tricky... But what I mean is, it is not "omg kernel threads are hard" that is the primary factor preventing direct access to them. It's that the language runtime has a lot of baggage.
If I use uWSGI is problem gone?
Current python "green threads" use the keywords async/await as an api. Underneath this api, the state of the art implementations use libuv (wrapped in a python library called uvloop) the exact SAME C++ library that powers nodejs.
Yes, Python has a GIL. Yes, lightweight threads are mostly good for IO bound tasks. Yes it can still be used effectively if you design your app correctly.
I'm sure there's some useful information in here, but it's not worth digging through the patronization to find it.
I wouldn't read this if it was a tutorial and not a story.
Statistically speaking, maybe that's the same as me not reading it at all in 90% of universes.
I wish she would put a review at the end of sources / main concepts so I can learn more.
This one, at least, is new.
According to the post, "the thing" with Python/Gunicorn/Gevent is it's less performant than one would like in some circumstances, and a lazy developer might tell you you need to 'set your system to "elastically scale up" at some pitiful utilization level, like 25-30% of the entire machine'.
That's probably true! But not that helpful if you don't say what those circumstances are. There are many circumstances where Python is appropriate for a web service, and many circumstances where green threads work just fine. Tell me when I need to consider using the more complicated solution, don't tell me the simpler solution is always useless and doomed from the start.
And jeez what do you make of the BOFH if Rachels style upsets you.
tl;dr: gunicorn doesn't know how to multiplex listeners and green threads will ruin your request latency.
The thing is, that post isn't very useful nor interesting. The point here is that the "simple" python architecture doesnt scale well at all so you might not want to use it if you're planning on scaling ever.
Seeing this comment at the top of the page on the highest ranked post on the HN front page for the only woman programmer that I am personally aware of who regularly makes the front page really feels like a kick in the gut and looks like sexist garbage. And I would like to think better of HN than that.
I've seen similar comments leveled at PG [1] and Zed Shaw [2], so I don't think it's just sexism.
[1] https://idlewords.com/2005/04/dabblers_and_blowhards.htm
However, what seems to have happened here is that a bunch of folks are upvoting this link to the top of HN because of who the author is.
Meanwhile, other HN readers find this particular post to be a waste of time because, frankly, the content of the post itself is not particularly interesting or useful for most HN readers. Other posts[1][2] by this author have been much better suited for the top of HN, for example
[1] https://rachelbythebay.com/w/2018/04/23/pace/ [2] https://rachelbythebay.com/w/2019/11/18/oldbash/
Basically she's saying that python async (which the current state of the art implementation uses libuv the same thing driving nodejs and consequently suffers from the same "problems") doesn't have actual concurrency. Computations block and concurrency only happens under a very specific case: IO. One computation can happen at a time with several IO calls in flight and context switching can only happen when an IO call in the computation occurs.
She fails to see why this is good:
Python async and nodejs do not need concurrency primitives like locks. You cannot have a deadlock happen under this model period. (note I'm not talking about python threading, I'm talking about async/await)
This pattern was designed for simple pipeline programming for webapps where the webapp just does some minor translations and authentication then offloads the actual processing to an external computation engine (usually known as a database). This is where the real processing meat happens but most programmers just deal with this stuff through an API (usually called SQL). It's good to not have to deal with locks, mutexes, deadlocks and race conditions in the webapp. This is a huge benefit in terms of managing complexity which she completely discounts.
This is dangerously wrong and I would suggest that you reconsider the steps that got you to this understanding because something really important has been lost. It is absolutely critical to understand that deadlocks are not why you have locks. Correctness during concurrent operation is why you have locks. Deadlocks are a failure state when you do not have correctness during concurrent operation. So are things like double-increment and double-create.
Parallelism does not imply deadlocking, concurrency implies deadlocking, and both NodeJS and Python are concurrent runtime environments. And I can guarantee you that, with a little skull sweat, you can write a deadlock in NodeJS or Python. It is very easy. If you need some help, here's a trivial example (and ordinarily I wouldn't use a link shortener here but this one is hefty, it just goes to the Typescript playground): https://bit.ly/2Tvjyze
Also, as a concrete, real-world, yes-it-happens-here example of where locking is important, consider that I've recently built a dependency injection framework in NodeJS--tried to use others' first, but my situation isn't covered by existing ones--and had to resort to a mutex to avoid double creation of objects within a single lifecycle. Creation of objects within this lifecycle happens asynchronously--it has to, as the act of creating the objects might itself rely on asynchronous operations. So, if I were to have a diamond-dependency (A deps B and C, B and C dep D), I will non-deterministically, and based on the creation time of B and C, create either one or two instances of D. I rely upon a mutex, keyed upon the dependency being created, to ensure that this does not happen.
.
I would also submit that perhaps you should adopt a principle of charity and think real hard about whether your priors are correct before you start talking about what she "fails" to see. Rachel is one of those people who has Been Around and while I also have Been Around, I understand that Rachel has Been Around More and I probably should be listening more than I should be smarming at her.
Just a thought.
Ok technically you're right. I am completely wrong when I say it can NEVER happen.
But let's be real here, you introduced DEADLOCKING deliberately by introducing LOCKS and by doing context switching at weird places to make it happen. When nodejs came out one of the selling points was the lack of deadlocks and locks.
Case in point: there are no lock libraries in standard NodeJS.
Think about it, why is a LOCK needed here? Let's say you didn't have locks AT all. Wherever the heck you are the current Node task technically has what is equivalent to a LOCK on everything. Why? because all node instructions are atomic and single threaded. This is what replaces LOCKS in nodejs. Your code example is just strange. The only place where your example is relevant is if there was another process.
>But as a concrete example of where locking is important, consider that I've recently built a dependency injection framework in NodeJS--tried to use others' first, but my situation isn't covered by existing ones--
Probably because, again, nobody really programs using DP in node let alone context switching and adding made up locks in the middle of all these injections and constructions. Whatever you're doing is probably very unique or (maybe, I don't know your specific situation) a sign over engineering. DP is a very bad pattern and is one of the primary sources of technical debt in code (especially when it's over 2 layers deep and in a diamond configuration)... but that's another topic. Anyway...
What is the point of "acquiring" a lock if in node I have a "LOCK" on everything? It makes no sense, whatever it is you're doing I am almost positive that there is a simpler way of doing it. Either way the dependency chain makes it obvious which needs to be created first and what can be created concurrently. The below psuedo code should produce what you're looking for without locks and with equivalent concurrency which is one of the main selling points of single threaded async.
b, c = await runAsync([B, () => C(await D())])
a = await A(b, c)
Think about it, node includes high level functions for http but none for locks which are an even lower level concept than http. It must mean you aren't supposed to use locks in Node.
I will say you are technically right in the fact that a deadlock CAN happen. I was wrong in saying it can NEVER happen, but you have to realize that I have a point here. Your example is really going very very far out of the way to pull it off.
>I would also submit that perhaps you should adopt a principle of charity and think real hard about whether your priors are correct before you start talking about what she "fails" to see. Rachel is one of those people who has Been Around and while I also have Been Around, I understand that Rachel has Been Around More and I probably should be listening more than I should be smarming at her.
I was not smarming her, whatever smarming means, I am disagreeing with her just like you are disagreeing with me. There is NOTHING wrong with disagreeing with anybody. What is wrong is when you are proven wrong and you don't accept it. I accept that my statement of a deadlock NEVER happening in nodejs is categorically wrong.
"Being around" does not entitle you to anything. I hate it when people say this, nothing personal. Do you even know how long I've been around? Additionally, the overall main point of my post still stands, which you didn't even really address. I don't think Rachel gets the point of green threads. I think we can both agree I've made a strong point and maybe you should use your own charity principles on me.
Just a thought.
Any headline that starts with "we have to talk about" can be answered by the words "do we?"
Is this something people actually have problems with in practice? I did lots of python and ran into it once. It was quickly fixed after a raised issue. I feel like non-toy development just doesn't experience it.
But maybe that's my environment bubble only. Do people who do serious python development actually have problem with this?
I have personally made this mistake on two or three occasions where I conflated file/module behavior with class behavior because I wanted a python file to act like it was a bit more declarative. Unfortunately this leads to a world of eternal pain. You can work around it, but you should have made everything a python class and pretended like the files/modules don't exist or at least have staggeringly different semantics hiding behind that innocent little `.` operator. Python simply cannot support the desire to solve a problem in a certain way because of the structure of the problem and forces you into using its happy path patterns if you want it to work in slightly different run time contexts.
Two relevant posts from instagram engineering on the subject which suggest that best practices for avoiding these kinds of issues are non-obvious and easy to miss.
https://news.ycombinator.com/item?id=20708889 https://news.ycombinator.com/item?id=21284669
I don't see how this can be an issue with imports. You have two cases: you imported the module with MTYPE before or after fork. Before: they will compare fine. (unless IPC's involved) After: you transferred the dict with MTYPE through some kind of serialisation or shmem and you cannot compare identities - that's a property of IPC rather than something to do with python modules.
Anyway, what I meant in previous comment was the risk matrix view. Sure, this can lead to bugs scoring various points in severity, but does it score high on likelihood?
In adtech you send 204 responses a lot. The body is empty, just the headers. Headers like 'Server' and 'Date'. Apache won't let you turn Server off... 'security through obscurity' or some nonsense. Why do I need to tell an upstream server my time 50k times per second?
Zip it all up! Nope, that only applies to the body which is already empty.
Egressing traffic! A cloud provider's dream. I wonder what percentage of their revenue come from clients sending the Date header.
See: https://tools.ietf.org/html/rfc2616#section-10.2.5
Seems like what you are having trouble with is Apache, not HTTP 1.1.