undefined | Better HN

0 points8cvor6j844qw_d64mo ago0 comments

I'll be interested in the incident writeup since DNS is mentioned. It will be interesting in a way if it is similar to what happened at AWS.

0 comments

Insanity4mo ago

It's pretty unlikely. AWS published a public 'RCA' https://aws.amazon.com/message/101925/. A race condition in a DNS 'record allocator' causing all DNS records for DDB to be wiped out.

I'm simplifying a bit, but I don't think it's likely that Azure has a similar race condition wiping out DNS records on _one_ system than then propagates to all others. The similarity might just end at "it was DNS".

parliament324mo ago

That RCA was fun. A distributed system with members that don't know about each other, don't bother with leader elections, and basically all stomp all over each other updating the records. It "worked fine" until one of the members had slightly increased latency and everything cascade-failed down from there. I'm sure there was missing (internal) context but it did not sound like a well-architected system at all.

nijave4mo ago

>slightly increased latency

They didn't provide any details on latency. It could have been delayed an hour or a day and no one noticed

RajT884mo ago

Needs STONITH

kyrra4mo ago

https://isitdns.com/

cdr4204mo ago

It's always DNS

tempest_4mo ago

It is a coin flip, heads DNS, tails BGP

1 more reply

layer84mo ago

DNS has both naming and cache invalidation, so no surprise it’s among the hardest things to get right. ;)

dotancohen4mo ago

That's three of the hardest problems in CS ))

j / k navigate · click thread line to collapse