CrowdStrike Update: Windows Bluescreen and Boot Loops - https://news.ycombinator.com/item?id=41002195 - July 2024 (3590 comments)
Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.
- They don't do enough or the right kind of smoke tests.
- They don't do exponential-canary deployments with an ability to rollback, and instead just YOLO it.
- They don't appear to have a customer-side security / client platform team update approval gating change control process for software updates or for definitions (or whatever they use).
This is fundamentally laziness and/or incompetency.
From management - that couldn't care less about low visibility, low impact projects
The company that lobbied the hardest and paid the most in bribes got the contracts.
You notice how this outage affected hospitals and airlines? There is a strong tendency in software sales for industries to align around one or two leaders. Oh, American chose Crowdstrike? Maybe we at Delta should just do what they did. Or literally Delta hires the VP from American to be their CISO and he just does what he did before.
Vendor selection is hard and buyer's remorse is frequently hard to deal with once you've sunk cost into a migration.
This seems very amateurish for companies who regularly talk professionally to win said contracts , whether the best product or not.
My guess is C-suite, crisis consultants and lawyers are involved heavily so the actual engineering folks have little voice now in any communication and we get stuff like this.
And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it causes a BSOD and inability to boot.
and it's kind of understandable that channel files might seem safe to update constantly without oversight, but that's just assuming that the file that interprets the channel file isn't a bunch of dogshit code.
https://sre.google/workbook/canarying-releases/
Which starts with "a majority of incidents are triggered by binary or configuration pushes". The stats for config related failures is one link away at
https://sre.google/workbook/postmortem-analysis/
Where it says 31% of outages in 2010-2017 are caused by "configuration push".
https://x.com/perpetualmaniac/status/1814376668095754753?s=4...
https://x.com/ananayarora/status/1814269058088304760
The authors explain the coding error and coredump well, but I'm lost: Is the buggy code that they're describing the channel file, or some kernel code that consumes the channel file? Is there a way to tell?
If it controls the behavior of a computer, then it's code.
> and it's kind of understandable that channel files might seem safe to update constantly without oversight
Yeah, no, it's not. They pushed an update that crashed the majority of their Windows installed base in a way that couldn't be fixed remotely. It doesn't matter what the update was to. It needed to be tested. There is no way that any deployment pipeline that could fail to catch something that blatant could possibly be "understandable".
... and that kernel mode code shouldn't have been parsing anything with any complexity to begin with. And should have been tested into oblivion, and possibly formally verified.
This is amateur-hour nonsense. Which is what you expect from most of these "Enterprise Cyber Security(TM)" vendors.
... AND the users shouldn't have just gone and shoved that kind of thing into every critical path they could think of.
I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation until this one outage.
You can't take days testing gradual roll outs for this type of content, because that's how long customers are left unprotected by that content. Although the root cause is on the channel files, I feel like the driver that processes them should have been able to handle the "logic bug" in question so we'll find out more over time I guess.
For example, with windows defender which runs on virtually all windows systems, the signature updates on billions of devices are pushed immediately (with exception to enterprise systems, but even then there is usually not much testing on signature files themselves, if at all). As far as the devops process Crowdstrike uses to test the channel files, I think it's best to leave commentary on that to actual insiders but these updates happen several times a day sometimes and get pushed to every Crowdstrike customer.
I certainly don't want to know (through disaster news) about the construction company that built the bridge I drive through everyday, not for another 15 years, not ever!
This kind of software simply should not fail, with such a massive install base on so many sensitive industries. We're better than that, the software industry is starting to mature and there are simple and widely-known procedures that could have been used to prevent it.
I have no idea how CrowdStrike stock has only dropped 10% to the values of 2 months ago. Actually, if the financial troubles you get into are only these, take back what I said, software should be failing a lot (why spend money on robustness when you don't lose money on bugs?)
If you can't take days to do it then do a gradual rollout in hours. It's not a high bar.
They certainly run their software on those many customers' systems, but but based on my experience with them, "protect" isn't a descriptor I'm willing to grant them.
We don't have the counter-factual where Crowdstrike doesn't exist, but I'm not convinced that they've been a net economic or security benefit to the world over the span of their existence.
I actually don't think it's outrageous that these files are rolled out globally, simultaneously. I'm guessing they're updated frequently and _should_ be largely benign.
What stands out to me is the fact that a bad config file can crash the system. No rollback mechanism. No safety checks. No safe, failure mode. Just BSOD.
Given the fix is simply deleting the broken file, it's astounding to me that the system's behavior is BSOD. To me, that's more damning that a bad "software update". These files seem to change often and frequently. Given they're critical path, they shouldn't have the ability to completely crash the system.
Anyone competent that manages software at scale should generally hold the opposite opinion to this.
The lesson of "multiple parser implementations for the same thing seems bad" and "sanity checks to prevent breaking things are hard heuristics to define" such that further changes were deferred.
All that to say that I can appreciate circumstances in which satisfying "don't crash the system" in response to configuration data can actually be fairly hard to realize. It can very significantly depend on the design of the pieces in question. But I also agree that it's pretty damning.
FWIW, at least Microsoft still "dogfoods" (and it's what coined that term), and even if the results of that aren't great, I'm sure they would've caught something of this severity... but then again, maybe not[1].
Push update to machines, observe, power cycle them, observe...
I could understand error in some rarer setup, but this was so common that it should have been obvious error.
Everyone has a buggy release at some point, but impacting global customers at this level is damn near unforgivable.
Heads need to roll for this oversight.
I don't understand CrowdStrike's rollout system, but given that people started seeing trouble earlier in the day, surely by that time they could have shut down the servers that were serving the updates, or something??
He also told me that soon after that the street outside the bank (another bank across the street, a hospital several blocks down) was lined with police who started barring entry to the buildings unless people had bank cards. By the time I woke up this morning technical people already knew basically what was going on, but I really underestimated how freaked out the average person must have been today.
The obvious joke here is CS runs the malicious C2 framework. So the system worked as designed: it prevented further execution and quarantined the affected machines.
But given they say that’s just a configuration file (then why the hell is it suffixed with .sys?), it’s actually plausible. A smart attacker could disguise themselves and use the same facilities as the CS. CS will try to block them and blocks itself in the process?
Given that this incident has now happened twice in the space of months (first on Linux, then on Windows), and that as stated in this very post the root cause analysis is not yet complete, I find that statement of “NO RISK” very hard to believe.
I’d like more information on how these Channel Files are created, tested, and deployed. What’s the minimum number of people that can do it? How fast can the process go?
> Although Channel Files end with the SYS extension, they are not kernel drivers.
OK, but I'm pretty sure usermode software can't cause a BSOD. Clearly something running in kernel mode ate shit and that brought the system down. Just because a channel file not in kernel mode ate shit doesn't mean your kernel mode software isn't culpable. This just seems like a sleezy dodge.
(Maybe there's some subtext that I'm missing, but I don't see how saying "these aren't kernel drivers" makes them look any better, and I do see why they might say it to be informative, so it looks like to me like they're doing the latter.)
It absolutely reads like this. They are getting blasted online for shipping kernel mode driver updates without proper QA and release engineering. Which just from face value just seems like some insano style engineering. They are saying "it's not actually a kernel mode value" to deflect blame.
I mean, I really don't understand why they would make this statement otherwise. If they are innocently just trying to say "this is just a channel file", there are other ways to say this, and it really isn't relevant enough to underline and emphasize.
> We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred.
There's always going to be flaws in the logic of the code, the trick is to not have single errors be so catastrophic.
How a common bug was rolled out globally with no controls, testing, or rollback strategy is the right question
And then absolutely the release process.
Rollback is hard I guess once your OS can't boot.
That's going to find a cause: a programmer made an error. That's not the root of the problem. The root of the problem is allowing such an error to be released (especially obvious because of its widespread impact).
Linux provides eBPF and macOS provides system extensions.
I'll also add that Windows itself heavily prioritizes backwards-compatibility over security, which leads companies to seek out third-party solutions for stopping malware instead of design-based mitigations being built into Windows.
And I'm not sure epbf actually allows you to do a lot of the stuff crowdstrike-like software does. I know they use it on Linux though so maybe eBPF has evolved a lot since I last looked at it.
Must be corrected to "the issue is not the result of or related to a cyberattack by external agents".
Very weak and over corporate level of ass covering. And it doesn't even come close to doing that.
They should just let the EM of the team involved provide a public detailed response that I'm sure is floating around internally. Just own the problem and address the questions rather than trying to play at politics, quite poorly.
https://www.nathanhandy.blog/images/blog/OSI%20Model%20in%20...
If the initial root cause analysis is correct, Crowdstrike has pushed out a bug that could have been easily stopped had software engineering best practices been followed: Unit Testing, Code Coverage, Integration Testing, Definition of Done.
If I ever get a sales pitch from these shit brains, they will get immediately shut down.
Also fuck MS and their awful operating system that then spawned this god awful product/company known as “CrowdStike Falcon”
By the way, Falcon can be and is deployed to Linux and MacOS hosts in these organisations too it's just that this particular incident only affected Windows.
1. critical infrastructure around the globe seemed to depend on CrowdStrike
2. "If I ever get a sales pitch from..." suggested you are in an environment that is far from critical infrastructure.
It also wouldn't happen on Linux: they use eBPF there which was designed by grownups and validates its inputs.
