This gives attackers an advantage (they are incentivized to read commits and can easily see the vuln) and defenders a huge disadvantage. Now I have to rush to patch whereas attackers have had this entire time to build their POCs and exploit systems.
End this ridiculous practice.
If you wish to disagree with how we handle all of this, wonderful, we will be glad to discuss it on the mailing lists. Just don't try to rehash all the same old arguments again, as that's not going to work at all.
Also, this was fixed in a public kernel last week, what prevented you from updating your kernel already? Did you need more time to test the last release?
Edit: It was fixed in a public release 12 days ago.
> Just don't try to rehash all the same old arguments again, as that's not going to work at all.
No shit. People have been trying to explain all of this to you for decades lol I'm not stupid enough to think I'll succeed where they've failed.
> Linus summarized the reasoning behind this behavior in an email to the Linux Kernel mailing list in 2008 ...
Since severity can be a moving target, it seems like there is no straightforward solution. With that said, by hiding the known ones, older distros don't have much of a hope in hell of getting all reported CVE fixes back-ported.
Why isn't there a public index mapping known CVE fixes to git commit IDs? This seems totally doable and would make the world a more secure place overall.
If you do not want to rush to patch more than you have to, use a LTS kernel and know that updates matter and should be applied asap regardless of the reason for the patch.
When someone submits a patch for a vulnerability label the commit with that information.
> You have to rush to patch in any case.
The difference is how much of a head start attackers have. Attackers are incentivized to read commits for obfuscated vulns - asking defenders to do that is just adding one more thing to our plates.
That's a huge difference.
> the logical step is that it doesn't require immediate action when the label is not there.
So I can go about my patch cycle as normal.
> Never mind that the bug might actually be exploitable but undiscovered by white hats.
OK? So? First of all, it's usually really obvious when a bug might be exploitable, or at least it would be if we didn't have commits obfuscating the details. Second, I'm not suggesting that you only apply security labeled patches.
Well Xen for instance includes a reference to the relevant security advisory; either "This is XSA-nnn" or "This is part of XSA-nnn".
> If the maintainers start to label commits with "security patch" the logical step is that it doesn't require immediate action when the label is not there. Never mind that the bug might actually be exploitable but undiscovered by white hats. If you do not want to rush to patch more than you have to, use a LTS kernel and know that updates matter and should be applied asap regardless of the reason for the patch.
So reading between the lines, there are two general approaches one might take:
1. Take the most recent release, and then only security fixes; perhaps only security fixes which are relevant to you.
2. Take all backported fixes, regardless of whether they're relevant to you.
Both Xen and Linux actually recommend #2: when we issue a security advisory, we recommend people build from the most recent stable tip. That's the combination of patches which has actually gotten the most testing; using something else introduces the risk that there are subtle dependencies between the patches that hasn't been identified. Additionally, as you say, there's a risk that some bug has been fixed whose security implications have been missed.
Nonethess, that approach has its downsides. Every time you change anything, you risk breaking something. In Linux in particular, many patches are chosen for backport by a neural network, without any human intervention whatsoever. Several times I've updated a point release of Linux to discover that some backport actually broke some other feature I was using.
In Xen's case, we give downstreams the information to make the decisions themselves: If companies feel the risk of additional churn is higher than the risk of missing potential fixes, we give them the tools do to so. Linux more or less forces you to take the first approach.
Then again, Linux's development velocity is way higher; from a practical perspective it may not be possible to catch the security angle of enough commits; so forcing downstreams to update may be the only reasonable solution.
(This is not a rhetorical question. I can possibly influence this policy, but unsubstantiated objections won’t help.)
1. The commit message [1] does not mention any security implication. This is reasonable, because the patch is usually released to the public earlier and it makes sense to do some obfuscation, to deter patch-gappers. But note that this approach is not a controversy-free one.
2. But there is also no security announcement in stable release notes or any similar stuff. I don't know how to provide evidence of "something simply does not exist".
3. Check the timeline in the blog post. The bug being fixed in stable release (5.6.11 on 2022-02-23) marks the end of upstream's handling of this bug. Max then had to send the bug details to linux-distros list to kick off (another separate process) distro maintainers' response. If what you are maintaining is not a distro, good luck.
Is this wrong-sounding enough?
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
Evidence of this being helpful to attackers and not defenders? IDK, talk to anyone who does Linux kernel exploit development.
edit: There you go, Greg linked his policy, which explicitly notes this.
Currently the situation is that you can just follow development/stable trees right (e.g. [0])? Why would you only want the security patches (of which there look to be a lot just in the last couple weeks). Are you looking to not apply a patch because LK devs haven't marked it as a security patch?
[0]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
If the patch was flagged as a security problem from the beginning, it would give advantage to attackers, since they would know that the particular patch is worth investigating, while the defenders would have to wait for the patch to be finalized and tested anyway.
Their point is that a full-time attacker (and there's enough money in it to do it as a full-time job these days) can look for obfuscated commits and take the time to deobfuscate them, whereas a defender doesn't have that kind of time.
Attackers don't just crawl code at random. Starting with known crashes or obfuscated commits is always faster.
To me, it seems like the average corporate security team is not going to worry about these kinds of attackers. Security for state secrets might, but they seem likely to be clued in early by Linux developers.
I'm probably missing something tho.
Virtually every single Linux user. I think what you're missing is how commonplace and straightforward it is for attackers to review these commits and how uncommon it is for someone to be on the receiving end of an embargo.
Most exploits are for N days, meaning that they're for vulnerabilities that have a patch out for them. Knowing that there's a patch is universally critical for all defenders.
For context, my company will be posting about a kernel (then) 0day one of our security researchers discovered. You can read other Linux kernel exploitation work we've done here: https://www.graplsecurity.com/blog
Really impressive debugging too.
I'm not a fan of such a policy. That usually leads to people zero-initializing everything. For this bug, this would have been correct, but sometimes, there is no good "initial" value, and zero is just another random value like all the 2^32-1 others.
Worse, if you zero-initialize everything, valgrind will be unable to find accesses to uninitialized variables, which hides the bug and makes it harder to find. If I have no good initial value for something, I'd rather leave it uninitialized.
So use a language that has an option type, we've only had them for what, 50 years now.
But I think it would be troublesome to use such a hypothetical feature in C if it's only available in some compiler-specific dialect(s), because you need to coerce to any type, so it would be hard to hide to hide behind a macro. What should it expand to on compilers without support? It would probably need lots of variants specific to scalar types, pointer types, etc., or lots of #if blocks, which would be unfortunate.
Zig is a nice language with this feature, and it fits into many of the same use cases as C: https://ziglang.org/documentation/0.9.1/#undefined
The issue is with dynamic memory allocation as that would be the responsibility of the allocator (and of course the kernel uses custom allocators).
The fix:
+++ b/lib/iov_iter.c
@@ -414,6 +414,7 @@ static size_t copy_page_to_iter_pipe(struct page \*page, size_t offset, size_t by
return 0;
buf->ops = &page_cache_pipe_buf_ops;
+ buf->flags = 0;
get_page(page);
buf->page = page;
buf->offset = offset;I'm a C++ guy, and the lack of constructors is one of many things that bothers me with C.
This proposal sounds great until you find out that this is a hard problem to solve reasonably well in the compiler and no matter what you do there will be valid programs that your compiler will reject.
> valid programs that your compiler will reject.
Do you mean "valid programs where everything is initialized when an object is created will somehow fail to detect that"?
VFS/Page Cache/FS layers represent incredible complexity and cross dependencies - but the good news is code is very mature by now and should not see changes like this too often.
One of the prevailing features of well driven open-sources project is - you're encouraged to improve the code i.e. make it better {readable, maintainable, faster, hard}. You're not encouraged to change it for the sake of change i.e impress people.
I've the feeling it is the first case because it reduced the number of lines and kept source readable. Aside from that, I don't think good developers want to impress others.
(from the downvotes it seems like some people don't want software to be a science)
I'd like to add, for the less tenured developers around: "with the level of experience you're just going to live the fact that there'll always be bugs."
The GCC version requirement is 5.1 (which is 7 years old). Before that, it was 4.9, 4.8, 4.6 and 3.2. It has never been 4.7.
Use of newer versions of C than C89 which provides solutions to actual issues is perfectly fine. C11 was picked because it does not require an increase in minimum GCC version to use it, making your entire argument pointless.
The Linux kernel is already pretty lenient, as many alternatives have their a compiler on the tree and target only that.
By this logic why not write the entire kernel in assembly? Tools evolve and improve over time and it makes sense to migrate to better tools over time. We shouldn't have to live in the past because you refuse to update your compiler.
> Blaming the Linux kernel (i.e. somebody else’s code) for data corruption must be the last resort.
^^^ I can only image the stress levels at this point.
Circa 10 lines of C. Beautiful.
First it helps to triage whether the bug is in your own code or in other code.
You also then can use it like the author and find the commit which introduced the bug.
And you can use it as a test case to verify if the bug is closed.
wish more would be written like this
Every few months, someone would always come up with a reason why "bad entropy" was the cause of a bug.
It never was, but it felt like a thing we had to go through, to get to the real cause.
I get responsible disclosure is important, but should we not give people some more opportunity to patch, which will always take some time?
Just curious.
Also, nice work and interesting find!
It puts me, as a defender, at an insane disadvantage. Attackers have the time, incentives, and skills to look at commits for vulns. I don't. I don't get paid for every commit I look at, I don't get value out of it.
This backwards process pushed by Greg KH and others upstream needs to die ASAP.
The announcement only serves to let the rest of the public know about this and incentivize them to upgrade.
(Thanks Max for handling this well and politely and for putting up with everyone’s conflicting opinions.)
I can't help but physically shake my head as I write this. I can't imagine actually asking people to try to play pretend security through obscurity because folks still can be arsed to implement some sort of reasonable update strategy. I have enough experience in tiny and huge shops to say that it's a matter of prioritization and it's just a blatant form of technical debt and poor foresight.
https://lists.debian.org/debian-security-announce/2022/msg00...
21.10 appears to be lacking the patch.
Luckily, almost all (if not just all) these millions of devices which will never be updated never ever received the vulnerable version in the first place. The bug was only introduced in 5.8 and due to how hardware vendors work phones are still stuck in 4.19 ages (or better, 5.4. but no 5.10 besides Pixel 6)
Based on the description, sounds like it should be quite possible.
Please update your devices ASAP!
Does it need patching? Of course. It’s not a privilege escalation remote code execution issue though, and even if it was, it would be on a tiny fraction of running devices right now.
That's correct and I misjudged the situation. Sorry!
Sorry!
That doesn't sound right.
> if I recall correctly, you just chop off the GZIP header.
...to get the raw DEFLATE stream, that is. You still need to attach any necessary metadata for PKZIP, which Max mentions. Their approach for converting between the two is pretty clever: it's so elegant and simple that it seems obvious, but I never would have thought of it. Very nifty, @max_k!
I mean, compiling 17 kernels alone takes so long that most people would've given up in between.
Nah, that's the most fun part. Once you have one kernel that works and one that doesn't, you can be pretty sure that you'll eventually find the cause of the bug. The part where I would have given up is the "trying to reproduce" part.
The real deal was tracking it down and creating a reproducible test.
But yeah, incredibly impressive persistence.
I was working in MSFT back than and I was writing a tool that produced 10 GB of data in TSV format , that I wanted to stream into gzip so that later this file would be sent over the network. When the other side received the file they would gunzip it successfully, but inside there would be mostly correct TSV data with some chunks of random binary garbage. Turned out that pipe operator was somehow causing this.
As a responsible citizen I tried to report it to the right team and there I ran into problems. Apparently no one in Windows wants to deal with bugs. IIt was ridiculously hard to report this while being an employee, I can't imagine anyone being able to report similar bugs from outside. And even though I reported that bug I saw no activity in it when I was leaving the company.
However I just tried to quickly reproduce it on Windows 10 and it wouldn't reproduce. Maybe I forgot some details of that bug or maybe indeed they fixed this by now.
Glad it looks like they got around to it though.
There are lots of things which have been fixed in Windows 10, I'd go so far to say 1903 (19H1) is where things started to settle down, but even the latest versions are not perfect. When the Israeli/Palestinian conflict broke out in 2019, some of the US military computers started playing up for about a week, after the US vetoed something at the UN level regarding this conflict. So MS still has a long way to go to get things secure.
What are the memory savings of this splicing approach as compared to streaming [through userspace]?
If the application were processing the data in some way, then it would be worth it. Otherwise it is better to skip all of that work.
It was a random, intermittent file corruption that didn't cause real harm to the authors organization and was, clearly, very tricky to track down.
Writing [repeated, if needed] monotonically increasing counters like this is a really good testing technique.
Via HTTP, all access logs of a month can be downloaded as a single .gz file. Using a trick (which involves Z_SYNC_FLUSH), we can just concatenate all gzipped daily log files without having to decompress and recompress them, which means this HTTP request consumes nearly no CPU. Memory bandwidth is saved by employing the splice() system call to feed data directly from the hard disk into the HTTP connection, without passing the kernel/userspace boundary (“zero-copy”).
Windows users can’t handle .gz files, but everybody can extract ZIP files. A ZIP file is just a container for .gz files, so we could use the same method to generate ZIP files on-the-fly; all we needed to do was send a ZIP header first, then concatenate all .gz file contents as usual, followed by the central directory (another kind of header).
Just want to say, these people are running a pretty impressive operation. Very thoroughly engineered system they have there.
Exciting for the implications of opening many locked down consumer devices out there.
Nightmare for the greater cyber sec world...
What a supérioritégrandeur!