The solution proposed in this post doesn't work, though: if the accept completes before the SQE for the cancellation is submitted, the FD will still be leaked. io-uring's async cancellation mechanism is just an optimization opportunity and doesn't synchronize anything, so it can't be relied on for correctness here. My library could have submitted a cancellation when the future drops as such an optimization, but couldn't have relied on it to ensure the accept does not complete.
Rust assumes as part of its model that "state only changes when polled". Which is to say, it's not really "async" at all (none of these libraries are), it's just a framework for suspending in-progress work until it's ready. But "it's ready" is still a synchronous operation.
But io-uring is actually async. Your process memory state is being changed by the kernel at moments that have nothing to do with the instruction being executed by the Rust code.
Still in Rust community "safety" is used in a very specific understanding, so I don't think it is correct to use any definition you like while speaking about Rust. Or at least, the article should start with your specific definition of safety/unsafety.
I don't want to reject the premise of the article, that this kind of safety is very important, but for Rust unsafety without using "unsafe" is much more important that an OS dying from leaked connections. I have read through the article looking for rust's kind of unsafety and I was found that I was tricked. It is very frustrating, it looks to me as a lie with some lame excuses afterwards.
But this specific case is not like that, my issue with the headline that it is a clickbait, and the article is written in a way, that you can't detect it early.
> anyone should do what they want in the end including saying this
I disagree. Any community, any group, any society has some rules, that exist for the good of the group. One of the most frequent rules is "do not lie". The headline de facto lies about the content of the article. I was mislead by it, and I'd bet that the most of rustaceans were misled also. Moreover, it seems to me, that it was a conscious manipulation by the author. Manipulation is also for the most groups of people is a bad thing, even if it doesn't rely on lies. It is bad not just for rustaceans. Here on HN you can see complaints about clickbaity titles very often, people are really annoyed by them. I'm relatively tolerant to clickbaits, but this was too much.
Containerd maintainers soon followed Google recommendations and updated seccomp profile to disallow io_uring calls [2].
io_uring was called out specifically for exposing increased attack surface by kernel security team as well long before G report was released [3].
Seems like less of a rust issue and more of a bug(s) in io_uring? I suppose user space apps can provide bandaid fix but ultimately needs to be handled at kernel.
[1] https://security.googleblog.com/2023/06/learnings-from-kctf-...
I'm working with io_uring currently and have to disagree hard on that one; io_uring definitely has issues, but the one here is that it's being used incorrectly, not something wrong with io_uring itself.
The io_uring issues overall also disaggregate in mostly 2 overall categories:
- lack of visibility into io_uring operations since they are no longer syscalls. This is an issue of adding e.g. seccomp and ptrace equivalents into io_uring. It's not something I'd even call a vulnerability, more of a missing feature.
- implementation correctness and concurrency issues due to its asynchronicity. It's just hard to do this correctly and bugs are being found and fixed. Some are security vulnerabilities. I'd call this a question of time for it to get stable and ready but I have no reason to believe this won't happen.
Being ALLOWED to be used badly, is the major cause of unsafety.
And consider that all the reports you reply to are by serious teams. NOT EVEN THEM succeed.
That is the #1 definition of
> something wrong with io_uring itself
It's such a shame. The io_uring interface is one of the coolest features of the kernel. It's essentially an asynchronous system call interface. A sane asynchronous I/O interface that finally did it right after many tries. Sucks to see it just get blocked everywhere because of vulnerabilities. I hope whatever problems it has get fixed because I want to write software that uses io_uring and I absolutely want to run said software on my phone with Termux.
The article is more about the Rust io_uring async implementation breaking assumption that Rust's async makes, in that a Future can only get modified when it's poll()-ed.
I'm guessing that assumption came from an expectation that all async runtimes live in userland, and this newfangled kernel-backed runtime does things on its own inside the kernel, thus breaking the original assumption.
Rust went with poll-based API and synchronous cancellation design anyway, because that fits ownership and borrowing.
Making async cancellation work safely even in presence of memory leaks (destructors don't always run) and panics remains an unsolved design problem.
The title of this submission is aimed specifically at "Async Rust", ie the language. The reality is that one third-party library with 4k stars on GitHub and 0.2.4 version number has a non-memory-unsafe leak.
I'd say the title is a full-on lie.
Now let's take "the future" part... you seem to be impugning Async Rust for something it's not even purported to do in the present. What's the point of this?
You found a bug in monoio it seems... I don't see the argument you've presented as supporting the premise that "Async Rust is not safe".
It's really that Rust might've made a poor choice here, as the article points out:
> Async Rust makes a few core assumptions about futures:
> 1. The state of futures only change when they are polled.
> 2. Futures are implicitly cancellable by simply never polling them again.
But at the same time, maybe this is just that Rust's Futures need to be used different here, in conjunction with a separate mechanism to manage the I/O operation that knows things need to be cancelled?
> We just need to encourage async runtimes to implement this fix.
This likely needs async drop if you need to perform a follow up call to cancel the outstanding tasks or closing the open sockets. Async Drop is currently experimental:
Maybe cancellation itself is problematic. There’s a reason it was dropped from threading APIs and AFAIK there is no way to externally cancel a goroutine. Goroutines are like async tasks with all the details hidden from you as it’s a higher level language.
Cooperative cancellation can be implemented in languages that mark their suspension points explicitly in their coroutines, like Rust, Python and C++.
I think Python's asyncio models cancellation fairly well with asyncio.CancelledError being raised from the suspension points, although you need to have some discipline to use async context managers or try/finally, and to wait for cancelled tasks at appropriate places. But you can write your coroutines with the expectation that they eventually make forward progress or exit normally (via return or exception).
It looks like Rust's cancellation model is far more blunt, if you are just allowed to drop the coroutine.
You can only drop it if you own it (and nobody has borrowed it), which means you can only drop it at an `await` point.
This effectively means you need to use RAII guard objects like in C++ in async code if you want to guarantee cleanup of external resources. But it's otherwise completely well behaved with epoll-based systems.
I find that a bigger issue in my async Rust code is using Tokio-style async "streams", where a cancelled sender looks exactly like a clean "end of stream". In this case, I use something like:
enum StreamValue<T> {
Value(T),
End,
}
This can obviously be wrapped. But any wrapper still requires the sender to explictly close the stream when done, and not via an implicit Drop.
FWIW, this was also painful to do across threads in our C event loop, but there was no way around the fact that we needed it (cf. https://github.com/FRRouting/frr/blob/56d994aecab08b9462f2c8... )
This pattern causes issues all over the place: in C++ with headaches around destruction failure and exceptions; in C++ with confusing semantics re: destruction of incompletely-initialized things; in Rust with "async drop"; in Rust (and all equivalent APIs) in situations like the one in this article, wherein failure to remember to clean up resources on IO multiplexer cancellation causes trouble; in Java and other GC-ful languages where custom destructors create confusion and bugs around when (if ever) and in the presence of what future program state destruction code actually runs.
Ironically, two of my least favorite programming languages are examples of ways to mitigate this issue: Golang and JavaScript runtimes:
Golang provides "defer", which, when promoted widely enough as an idiom, makes destructor semantics explicit and provides simple and consistent error semantics. "defer" doesn't actually solve the problem of leaks/partial state being left around, but gives people an obvious option to solve it themselves by hand.
JavaScript runtimes go to a similar extreme: no custom destructors, and a stdlib/runtime so restrictive and thick (vis-a-vis IO primitives like sockets and weird in-memory states) that it's hard for users to even get into sticky situations related to auto-destruction.
Zig also does a decent job here, but only with memory allocations/allocators (which are ironically one of the few resource types that can be handled automatically in most cases).
I feel like Rust could have been the definitive solution to RAII-destruction-related issues, but chose instead to double down on the C++ approach to everyone's detriment. Specifically, because Rust has so much compile-time metadata attached to values in the program (mutability-or-not, unsafety-or-not, movability/copyabiliy/etc.), I often imagine a path-not-taken in which automatic destruction (and custom automatic destructor code) was only allowed for types and destructors that provably interacted only with in-user-memory state. Things referencing other state could be detected at compile time and required to deal with that state in explicit, non-automatic destructor code (think Python context-managers or drop handles requiring an explicit ".execute()" call).
I don't think that world would honestly be too different from the one we live in. The rust runtime wouldn't have to get much thicker--we'd have to tag data returned from syscalls that don't imply the existence of cleanup-required state (e.g. select(2), and allocator calls--since we could still automatically run destructors that only interact with cleanup-safe user-memory-only values), and untagged data (whether from e.g. fopen(2) or an unsafe/opaque FFI call or asm! block) would require explicit manual destruction.
This wouldn't solve all problems. Memory leaks would still be possible. Automatic memory-only destructors would still risk lockups due to e.g. pagefaults/CoW dirtying or infinite loops, and could still crash. But it would "head off at the pass" tons of issues--not just the one in the article:
Side-effectful functions would become much more explicit (and not as easily concealable with if-error-panic-internally); library authors would be encouraged to separate out external-state-containing structs from user-memory-state-containing ones; destructor errors would become synonymous with specific programmer errors related to in-memory twiddling (e.g. out of bounds accesses) rather than failures to account for every possible state of an external resource, and as a result automatic destructor errors unconditionally aborting the program would become less contentious; the surface area for challenges like "async drop" would be massively reduced or sidestepped entirely by removing the need for asynchronous destructors; destructor-related crash information would be easier to obtain even in non-unwinding environments...
Maybe I'm wrong and this would require way too much manual work on the part of users coding to APIs requiring explicit destructor calls.
But heck, I can dream, can't I?
If explicit destruction is desirable, IMO the C#-style `using` pattern (try-with-resource in Java, `with` in Python etc) makes more sense than `defer` since it still forces the use of the correct cleanup code for a given type while retaining the explicitness and allowing linters to detect missing calls (because destructability is baked into the type). `defer` is unnecessarily generic here IMO.
In JS, you still end up having to write try/finally for anything that needs explicit resource management, so I don't think it's a good example. They are adding `using`, though, and TS already provides it.
I think that automatic and outside-of-written-code destruction is generally a risky/bad idea for nontrivial (in-user-memory only) objects. Both GC-ful languages' finalization systems and the automatic-destruction side of RAII are problematic.
I agree that moving more logic into explicit acquire/perform/destroy blocks like TWR and 'with' is good. I wish we had a language (rather than, say, just specific libraries) that could require such handling for certain datatypes.
That going to the sleep branch of the select should cancel the accept? Will cancelling the accept terminate any already-accepted connections? Shouldn't it be delayed instead?
Shouldn't newly accepted connections be dropped only if the listener is dropped, rather than when the listener.accept() future is dropped? If listener.accept() is dropped, the queue should be with the listener object, and thus the event should still be available in that queue on the next listener.accept().
This seems more like a bug with the runtime than anything.
Even the blog admits that cancellation of any kind, is racing with the kernel which might complete the accept request anyway. Even if you call `.cancel()`, the queue might have an accepted connection FD in it. Even if it doesn't, it might do by the time the kernel acknowledges the cancellation.
So you now have a mistakenly-accepted connection. What do you do with it? Drop it? That seems like the wrong answer, whoever writes a loop like the one in the blog will definitely not expect some of the connections mysteriously being dropped.
Having programmed in raw C, I know Rust is more like Typescript if you once try it after writing Javascript, you can't go back for anything serious in plain Javascript. You would want to have some guard rails better than having no guard rails.
Lifetimes are still used but not nearly as much.
Note that I don't know a lot about Rust, and I'm not familiar with the rules for Rust in the kernel, so it's possible that it's either not a problem or the problematic usages violate the kernel coding rules. (although in the latter case it doesn't help with non-kernel frameworks like SPDK)
Edit: I realize my comment might come off as a bit snarky or uninformative to someone who isn't familiar with Rust. That was not the intention. "Async Rust" is particular framework for abstracting over various non-blocking IO operations (and more). It allows terse code to be written using a few convenient keywords, that causes a certain state machine (consisting of ordinary Rust code adhering to certain rules) to be generated, which in turn can be coupled with an "async runtime" (of which there are many) to perform the IO actions described by the code. The rules that govern the code generated by these convenient keywords, i.e. the code that the async runtimes execute, are apparently not a great fit for io_uring and the like.
However, I don't think anyone is proposing writing such code inside the kernel, nor that any of the async runtimes actually make sense in a kernel setting. The issues in this article don't exist when there is no async Rust code. Asynchronous operations can, of course, still be performed, but one has to manage that without the convenient scaffolding afforded by async Rust and the runtimes.
lose?
Here by "async" I don't so much mean async/await versus threads, but these kernel-level event interfaces regardless of which abstraction a programming language lays on top of them.
At the 30,000 foot view, all the async abstractions are basically the same, right? You just tell the kernel "I want to know about these things, wake me up when they happen." Surely the exact way in which they happen is not something so fundamental that you couldn't wrap an abstraction around all of them, right?
And to some extent you can, but the result is generally so lowest-common-denominator as to appeal to nobody.
Instead, every major change in how we handle async has essentially obsoleted the entire programming stack based on the previous ones. Changing from select to epoll was not just a matter of switching out the fundamental primitive, it tended to cascade up almost the entire stack. Huge swathes of code had to be rewritten to accommodate it, not just the core where you could do a bit of work and "just" swap out epoll for select.
Now we're doing it again with io_uring. You can't "just" swap out your epoll for io_uring and go zoomier. It cascades quite a ways up the stack. It turns out the guarantees that these async handlers provide are very different and very difficult to abstract. I've seen people discuss how to bring io_uring to Go and the answer seems to basically be "it breaks so much that it is questionable if it is practically possible". An ongoing discussion on an Erlang forum seems to imply it's not easy there (https://erlangforums.com/t/erlang-io-uring-support/765); I'd bet it reaches up "less far" into the stack but it's still a huge change to BEAM, not "just" swapping out the way async events come in. I'm sure many other similar discussions are happening everywhere with regards to how to bring io_uring into existing code, both runtimes and user-level code.
This does not mean the problem is unsolvable by any means. This is not a complaint, or a pronunciation of doom, or an exhortation to panic, or anything like that. We did indeed collectively switch from select to epoll. We will collectively switch to io_uring eventually. Rust will certainly be made to work with it. I am less certain about the ability of shared libraries to be efficiently and easily written that work in both environments, though; if you lowest-common-denominator enough to work in both you're probably taking on the very disadvantages of epoll in the first place. But programmers are clever and have a lot of motivation here. I'm sure interesting solutions will emerge.
I'm just highlighting that as you grow in your programming skill and your software architecture abilities and general system engineering, this provides a very interesting window into how abstractions can not just leak a little, but leak a lot, a long ways up the stack, much farther than your intuition may suggest. Even as I am typing this, my own intuition is still telling me "Oh, how hard can this really be?" And the answer my eyes and my experience give my intuition is, "Very! Even if I can't tell you every last reason why in exhaustive detail, the evidence is clear!" If it were "just" a matter of switching, as easy as it feels like it ought to be, we'd all already be switched. But we're not, because it isn't.
Consider Linux async mechanisms. They are provided by a monolithic kernel that dictates massive swathes of what worldview a program is developed in. When select was found lacking, it took time for epoll to arrive. Then io_uring took its sweet time. When the kernel is lacking, the kernel must change, and that is painful. Now consider a hypothetical microkernel/exokernel where a program just gets bare asynchronous notifications about hardware and from other programs. Async abstractions must be built on top, in services and libraries, to make programming feasible. Say the analogous epoll library is found lacking. Someone must uproot it and perhaps lower layers and build an io_uring library instead. I will not say this is always less pain that before, although it is decidedly not the same as changing a kernel. But perhaps it is less painful in most cases. I do not think it is ever more painful. This is the essential pain brought about by stability and change.
"So I think this is the solution we should all adopt and move forward with: io-uring controls the buffers, the fastest interfaces on io-uring are the buffered interfaces, the unbuffered interfaces make an extra copy. We can stop being mired in trying to force the language to do something impossible. But there are still many many interesting questions ahead."
Having written a few libs for working with io_uring (in Ruby), cancellation is indeed tricky, with regards to keeping track of buffers. This is an area where working with fibers (i.e. stackful coroutines) is beneficial. If you keep metadata (and even buffers) for the ongoing I/O op on the stack, there's much less book-keeping involved. Managing the I/O op lifetime, especially cleaning up, becomes much simpler, as long as you make sure to not return before receiving a CQE, even after having issued a cancellation SQE.