Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC.
I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack
Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.
Just browse through the ./drivers tree of the kernel source some time and look at all the "quirks" and "workarounds" there. Recognize that virtually all of those could have been fixed in firmware, but weren't because no one cared.
It's just deeply depressing. This was a virtuoso debugging performance, but it didn't have to be that way if hardware companies were sane. But they aren't, and we all pay for it.
(edit: I just checked for myself, the Linux e1000e driver apparently doesn't support runtime firmware update via the kernel API, so linux-firmware wouldn't be expected to have this. I don't know what the process is for an affected end user to get a copy. I suspect there is none.)
This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development.
Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) that are not...
HIGHLY recommended to anyone who enjoyed the article.
It's a remote PHY injection at the hardware level - basically hacking the chip directly thanks to "special" content, i.e. putting a packet header within the packet.
Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way.
What's more interesting is that it bypasses all of the checks to prevent this from happening.
Here is the wiki page on the INVITE OF DEATH which sounds like the problem you hit:
Can you elaborate? I recognize the words, but not the meaning.
"Oh yeah, I used to work at Intel - that nic's got a YAML parser in it"…
For e.g. a NIC, it's not that many things that need to go wrong when you encapsulate a packet it should forward out on the wire so it rather looks like a control packet triggering some undesired results on the chip. Or vice versa.
Definitely not this bug, the one linked is not intel specific.
I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again.
This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab.
The idea is that you want to create a condition where you do see it, and the other handful of lockups that are there that you haven't yet seen.
Given that, for all we know the relevant parties did conduct extensive fuzz testing and your condescension is misplaced.
It wouldn't surprise me at all if there are other issues that don't depend on these exact circumstances to get something to fall over.
Since it's caused by a specific byte at a specific place, surely you'd only need to fuzz an average of 256 packets (of the required length) to find it... which suggests it wasn't done at all... zero...
There were 4 buggy bytes. 3 would crash the card. 1 would fix the card ("inoculating" it). If they only fuzzed once, or fuzzed with the same dataset (random.seed(1) // seed chosen at random), then they wouldn't catch it.
Even then, it's pretty sloppy.
A true fuzzing run for something like this ought to run in the tens of millions of packets.
If there's denial of service hiding in there I wonder about what other security bugs might be lurking. It's scary stuff, and pretty much impossible to audit yourself.
Edit:
Also, I'm a little freaked-out that the EEPROM on the NIC can be modified easily with ethtool. I would have hoped for some signature verification. I guess I'm hoping for too much.
Edit 2:
I wonder if this isn't the same issue described here: https://bugzilla.redhat.com/show_bug.cgi?id=632650
Depending on what functionality is being offloaded to the NIC (are there still NICs that do IPSEC and crypto offload?) there's the possibility for information disclosure vulnerabilities in the NIC itself. Yikes.
Generally the EEPROM does not contain anything like executable code, although it probably can contain patches for microcode on many NICs.
They spent a long time "showing" us that a different version of the Linux kernel didn't exhibit the problem so it must be a Linux kernel bug. Turned out the different version just sent data differently so it didn't trigger the same bug with the same data. Other data would have triggered it.
I wouldn't be surprised if the majority of "bugs" they receive reports on turn out to not be bugs in their hardware. There's probably parallels with the reports of compiler bugs, most end up not being bugs in the compiler.
The unfortunate truth is that responsibility of proving it's the vendors bug falls on the customer.
I had to write a proof-of-concept "exploit" to show the problem was with their hardware, effectively troubleshooting most of the problem for them.
It's always someone else's testing procedures, someone's else's hardware... The thing is though, most of the time it is. Tech support at the lower levels especially are used to dealing with people who have bad configurations are are using the products incorrectly. The annoyance comes in when you as a customer narrow a problem down but can't get anyone on the phone who can help you at that level.
We had problems when some NFS traffic would end up getting stalled. Our NFS server would use UDP packets larger than the MTU and they would end up getting fragmented.
Turns out the NIC would not look at the fragmentation headers of the IP packet and always assume a UDP header was present. From time to time, the payload of the NFS packet would have user data that matched the UDP port number the NIC would scan for to determine if the packet should be forwarded to the BMC. This motherboard had no BMC but it was configured as if it did have one.
It would time out after a second or so but in the meantime drop a bunch of packets. The NFS server would retransmit the packet but since the payload didn't change, the NIC would reliably drop the rest of the fragments of the packet.
Of course Intel claimed it wasn't their bug ("it's a bug in the Linux NFS implementation") but they quickly changed their tune when I coded up a sample program that would send one packet a second and reliably cause the NIC to drop 99% of packets received.
While it turned out to be a fairly lame implementation problem on Intel's part (both by ignoring the fragmentation headers and the poor implementation of the motherboard) I have to say it was very satisfying to solve the mystery.
Having the NIC inspecting incoming frames and potentially diverting them to the management controller sounds like a scary proposition. I'd almost rather just have dedicated Ethernet hardware for the management controller. The decrease in switch ports needed is certainly seductive, but I wonder if it's worth the risk.
(Do you happen to recall which Intel motherboard this bit you on? I was just getting out of whitebox Intel motherboard-based server builds about the time you're describing, but I'm just curious if only for the nostalgia.)
Yikes! Sounds like system management mode in a BIOS!
"With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death - and kill client machines behind firewalls!"
That's worrisome, I'll bet there are lots of not-so-nice guys trying to figure out a way to do just that. There must be tons of server hardware out there with these cards in them.
$ wget http://zorinaq.com/pub/intel-packet-of-death.txt
It has 0x32 at offset 0x47f regardless of the size of the IP and TCP headers. Try to run the wget AS SOON AFTER HAVING COLD BOOTED the machine (it is the very first packet of 1152+ bytes that determines if the NIC will crash or be inoculated until the next cold boot; well... unless it is the "no-op" packet).
Edit: fixed link.
Edit: Per http://www.kriskinc.com/intel-pod it does work on broadcast frames. Yikes!
You'd think the odds of getting a packet with 0x32 in position 0x47f is almost 1/256 per packet? So why aren't these network cards falling over everywhere every few seconds?
There are subtle differences between the offsets I get when I run "ethtool -e interface" versus those in the article that indicate an affected card (but they're quite close).
Mine are:
0x0010: ff ff ff ff 6b 02 69 83 43 10 d3 10 ff ff 58 a5
0x0030: c9 6c 50 31 3e 07 0b 46 84 2d 40 01 00 f0 06 07
0x0060: 00 01 00 40 48 13 13 40 ff ff ff ff ff ff ff ff
Output of "ethtool -i interface" (in case anyone wants to compare notes):
driver: e1000e version: 1.5.1-k firmware-version: 1.8-0
I tested both packet replays by broadcasting to all attached devices on a simple Gbit switch and no links dropped.
EDIT: it's difficult to tell definitively doing it remotely but I still can't re-produce the bug after a cold boot.
Correct me if I'm wrong (no, seriously) -- aren't both "packets of death" just poor handling of said malformed packets? Violations of their respective protocols? (TCP/SIP)
As a telecoms engineer predominantly selling Asterisk for the last 4 years and Asterisk experiance extending back to 2006 it's shocking to see this finally put right. For so many years, I have avoided the e1000 Intel controllers after a very public/embarassing situation when a conferencing server behaved in a wierd manner disrupting core services. Not having the expertise the author has, I narrowed it down to the Eth. Controller, Immediately replaced the server with IBM Hardware with Broadcom chipset and resumed our services in providing conferencing to some of the top FTSE100 companies.
Following this episode, I spend numerous days diagnosing the chipset with many conference calls with Digium engineers debugging the server remotely. In the end, no solution, recommendation to avoid the e1000 chipset and moved on.
# awk '/eth/ { print $1 }' <(ifconfig -a) | cut -d':' -f1 | uniq | while read interface; do echo -n "$interface "; ethtool -i $interface | grep driver; done
eth0 driver: e1000e
eth1 driver: e1000eThe linux e1000e may support many chipsets, so the fact that it's in service on your box doesn't necessarily mean you're running the suspect chipset, or that it's vulnerable.
Check with lspci -v, and check with the concrete test using the cold boot+magic packet others and the OP have posted.
Intel 82574L ethernet controller looks to be popular too. Intel, Supermicro, Tyan and Asus use it on multiple current motherboards and Asus notably on their WS (Workstation) variants of consumer motherboards, e.g. the Asus P8Z77 WS (socket LGA 1155) and Asus Z9PE-D8 WS (dual CPU, socket LGA 2011).
I suppose I can send just the right ICMP echo packet to router to make it send me back an innoculating frame.
One of the more painful summers resulted from a QLogic HBA which sometimes, for no apparent reason, injected a string of hex digits into the data it transmitted. There is a commemorative t-shirt of that bug with just the string of characters. It lead NetApp to putting in-block checksums into the file system so that corruption between the disk and memory, which was 'self inflicted' (and so passed various channel integrity checks) could be detected.
Here at Blekko we had a packet fragment that would simply vanish into the center switch. It would go in and never come out. We never got a satisfactory answer for that one. Keith, our chief architect, worked around it by randomizing the packet on a retransmit request.
The amount of code between your data and you that you can't control is, sadly, way larger than you probably would like.
/edit Ah, I spoke to soon; the author has updated his page here with diffs between affected and unaffected EEPROMs:
Sometimes bug fixing simply takes two people to lock themselves in a room and nearly kill themselves for two days.
Could this be a bug in the code of the EEPROM that handles TCP offloading, or one of the other hardware features that are now becoming more common? (https://en.wikipedia.org/wiki/TCP_offload_engine)
Basically one linux box with NVidia embedded gigabit controller could take down the whole segment. It would only happen after a random period, like after days when the box was busy. No two machines connected to the same switch would be able to ping each other any more after that. I suspected the switch, bad cables, etc. In the end I successfully circumvented the problem by buying a discrete gigabit ethernet card for the server in question.
He was instrumental in taming the Soekris and Alix SBC boards of old and creating Asterisk appliances with them. If you've got a little asterisk box running on some embedded looking hardware somewhere, it doesn't matter whose name is on the sticker, its got some Kielhofner in it.
I live about a mile from Star2Star. I ought to pop in one of these days and see what they're up to.
I'm currently working on an open source project where we are chasing "hang really hard and need a reboot to come back" issues with exactly this same ethernet controller, the Intel 82574L. I wonder if it's related!
Our Github issue: https://github.com/SnabbCo/snabbswitch/issues/39
Personally, I am not at all surprised that this sort of thing exists. I'm sure there's lots more defects out there to be found. turning completeness is a cruel master.
The “red telephone,” used to shut down the entire Internet comes to mind.
You discovered howto immunize friends and kill enemies in CyberWars.
Do governments have an Internet kill switch?
Yes, see Egypt & Syria they're good examples. We know China is doing Cyberwars, they are beyond Kill-Switches.
Techcrunch: http://techcrunch.com/2011/03/06/in-search-of-the-internet-k...
Wiki: http://en.wikipedia.org/wiki/Internet_kill_switch
We know Goverments deploy hardware that they can control when needed. Smartphones are the best examples for Goverment issued backdoors, next to some Intel Hardware (including NICs).