AWS North Virginia data center outage – resolved (opens in new tab)

AWS’s US-East 1 continues to be the Achilles heel of the Internet.

And while yes building across multiple regions and AZs is a thing, AWS has had a string of issues where US-East 1 has broader impacts, which makes things far less redundant and resilient than AWS implies.

Actual CNBC title: AWS data center outage hits trading on FanDuel, Coinbase - recovery to take hours

These things are dangerous. Someone who can take AWS down such as an employee can place a bet.

These bets aren’t as innocent as they seem because the bettors can often influence or change the outcome.

I thought cooling was pretty much pre-planned in any data center, and you simply don't install more stuff than you can cool?

So did some cooling equipment fail here or was there an external reason for the overheating? Or does Amazon overbook the cooling in their data centers?

I wonder if hetzner had better uptime in EU than AWS this year.

Related:

AWS EC2 outage in use1-az4 (us-east-1)

https://news.ycombinator.com/item?id=48057294

It's always East 1... Jokes aside I don't understand how often east-1 is taken down compared to other regions. Like it should be pretty similar to other regions architecture wise.

Coinbase claimed multiple AZs were down but the AWS statement was that only a single AZ was affected. Does anyone have more details?

So in the comments here we have the usual about us-east-1, it's centralized, it's a SPOF for AWS, they should fix it, don't put your stuff there, etc.

This was one data centre in one zone of a multi-zone region.

Yes IAM/R53 and others are centralized there, yes, reworking those service to be decentralized and cross-region would be a Good Thing. But us-east-1 is already multi-zone (6 with a seventh marked as "coming in 2026") with multi DC within zones. From memory, when a global service like IAM is out, it's more likely to be bugs in the implementation or dependency than a "if this was cross-region it wouldn't have died" issue.

But this wasn't an outage of any AWS global service this time. The only one that seemed to have more impact was/is MSK. Which is likely to be more of an issue with Kafka than anything AWS related.

I remember someone said friends dont let friends use USE1 last time and I thought that as the slack message saying USE1 and all the stuff we deploy there has gone to shit.

Could someone explain to me why they don't build these things near oceans? Like nuclear plants that need plenty cooling capacity too

Two loop cycle with heat exchanger to get rid of the heat

2/last 365 days down. My Ubuntu nas is 0/last 365days down.

Come and give me your cash if you want resilience.

There sure are a lot of eggs in that East basket.

using aws since s3 came out and i’ve yet to see any major company do multi az failover in any capacity whatsoever. default region ftw

How many nines of are we at this year?

In this thread: People confusing AZs, Regions, and why people would want to use single AZs.

Right, cooling.

both realtime markets where multi-AZ is hard?

I don't see anything on downdetector suggesting this was particularly disruptive.

Once known for having super reliable services, I've heard this company is scrambling to re hire some of the engineers they overconfidently "replaced" with AI.

When customers pay for cloud services, they expect them to be maintained by competent engineers.

edit: Not sure why the downvotes. If you fire the engineers that have been keeping your systems running reliably for years, what do you expect to happen?

I bet post-mortem will say vibe coding confused fahrenheit and celsius, we run too hot...

[flagged]

us-east-1 is down? shocking! stop putting SPOF services there. this location has had frequent issues for the past 15 years.