How four packets broke CenturyLink's network (opens in new tab)

It's kind of funny. These are best practices for running basic run of the mill web services, even something like a forum or personal homepage. Admittedly there's an obvious, massive difference in complexity, but you would expect the gold standard best practices to come from something mission critical like core Internet services and flow down to less critical services, not the other way around.

> How is this the first time this has come up?

It's not, obviously.

If one is cynical, it's just a way for the FCC to look like it is doing something. Or, if one attributes great, great rhetorical skill to the FCC, it's their way to lambaste CenturyLink for not even adhering to 101 level principles. I tend to believe the latter.

Here is a much more technical and thorough analysis of the incident: https://blog.thousandeyes.com/centurylink-outage-lessons-man...

The oddest part was that FFC recommendations didn’t include “limit the size of your broadcast domain”.

Large flat L2 are a classic time bomb, with their builders proudly exclaiming “look ma, no hands!” until they painfully get reminded of their mistake :)

I worked for a regional isp where almost the entire metro network was a single layer 2 network up to the point of presences. We regularly (as in at least once a week) had spanning tree loops and broadcast storms that took down the entire city.

Is there a source or is that a guess at what happened? That sounds immensely incompetent to the point that I find it hard to believe

I know some of these words.

I'd bet it's due to a firmware/software bug triggered by a rare condition, or undefined software behavior trigger by a hardware malfunction. If it's true, it means the root cause would probably never be identified as nobody can reproduce it. It's something pretty scary to think about: We can never guarantee most software would work correctly all the time, empirical testing is often the only practical assessment, and probabilistic bugs such as mysterious crashes cannot be discovered.

But I think the bigger problem is not the packets, but why didn't the backbone reject those malformed packets.

Assuming that by

> 3. no expiration time, meaning that the packet would not be dropped for being created too long ago; and

they mean the TTL was set to zero.

From RFC 1812:

> A router MUST NOT originate or forward a datagram with a Time-to-Live (TTL) value of zero.

So a packet with a TTL=0 should never be on the wire (Example a router receives a packet with TTL=1, if it's not destined for that specific router, then it gets discarded). My guess is the switching vendor had bad code that didn't handle TTL=0.

Ethernet.

Yes, combined with several amplification bugs, became a perfect [broadcast] storm

There was a footnote in the report about that:

> In the Bureau’s discussions with Infinera, Infinera used the term “packet” to describe what some experts refer to as Ethernet frames that are sent between nodes. For the sake of simplicity, this report uses the term “packet.”

We can remove that ambiguity by de-baiting the title and taking out "rotten".