AWS Service Interuptions

> What do you expect though, both companies are known for lying through teeth to convince the IT community (or more likely the IT managers) that their services are reliable for everyone and amazing uptime

Their uptime is much higher on average than any IT team I've ever been involved in.

Those icons don't change unless a certain percentage of the overall count of instances in an AZ or region are affected.

Most of the time people here might be seeing good portions of their infra go away, but the number isn't statistically significant to the overall region health for them to post an outage.

Don't ask me what those numbers are, but that is the way it is determined.

Sounds interesting. Is that data available?

From my experience in AWS, part of the problem is scope of impact. It's easy to lose track of just how many active customers there are at any time, and it's easy to see the platform as a cohesive whole, i.e. "If it's affecting you it must be affecting everyone else". In reality almost every customer impacting event affects only a tiny percentage of the active users at any one time. I know it can be hard to believe or see this as an external customer, because after all the service appears to be down to you. Take, for example, when people start saying "us-east-1a" is down. What is "us-east-1a"? If you've watched some of the re-invent talks you'll know that it actually describes numerous data centres, in close proximity (within a certain millisecond network target). If one of those has an incident, it might look to some customers like "us-east-1a" is down, when the reality might be that 95%+ of the data centres still fully functional, and most customers aren't seeing an impact.

You might have an incident affecting just 2% of the API calls, and affecting less than 2% of the user base (even that would be unusually large and a source of big drama internally). The service could be super stable and extremely reliable, but that 98% could get completely the wrong idea if they saw a service status, (and of course from a PR perspective, the same goes for anyone looking to use the platform.)

A service dashboard is an extremely blunt tool with which to pass out a message about service status. It renders what is an extremely nuanced situation down to "All good, maybe, no, DEAD"

To give a rough example, one service I was familiar with had a "page everyone in the team" level of incident. API availability tanked, badly. It looked atrocious, and seemed like hardly any requests were getting through successfully. You'd have every expectation that they should at least post a yellow alert, if not approaching red. It turned out that it was one single customer who's requests were failing (I forget the reason why), but due to a bug in the customer's software consuming the API, every time it got a 500 response, it would immediately resend the request, every single time, with no timeout or limited retry number. It reached such a terrific pace it got to the point where they made up a huge majority of all the requests hitting the endpoint. Every other customer using the service was completely fine. If you'd looked at the API graphs you'd think "POST YELLOW, POST YELLOW, NOW NOW NOW!", but because they took time to figure out the actual impact, they found out that would have been totally the wrong thing to do.

Service health dashboards are a neat idea, but one that is in desperate need of a rethink and overhaul. It has some value when you're a smaller service, but it just doesn't accurately scale with the platform.

I'm not sure what the real solution is. They've somehow got to pull together TB of logs and/or metrics to make an accurate assessment of the scenario, and do it in a matter of minutes, so as to provide accurate updates, and not needlessly panic customers.

This is the most concerning thing to me. The Multi-AZ, redundant setup is worthless if the ELB can't do its job properly. I've seen some funky behavior from the ELBs when it comes to instance state. They really need to make this better.

Availability zones are different from regions. You can be in multiple AZ's within the Sydney region still.

That's interesting. Is it an Australian regulation? Curious that they'd make it in such a way that the data had to reside in the same building/zone.

Wasn't quite that simple. I lost connectivity to instances that did not reboot so I'm guessing it took out some network elements.

Azure might be great in a year or so, but it makes me uneasy as is. Some of the services are great, but a lot of them are pretty fragmented. I've had so many instances where our billing/usage data has just "disappeared" for a few days, undocumented changes have been made to the formats of reports/exports/APIs, and official documentation is plain wrong that I just can't recommend Azure to anyone. Not to mention they have the most expensive infrastructure costs out of the major players (even with an EA and decent monetary commitment); their premium for Windows licensing is the lowest by far though (not surprising), so it does end up being a cheaper option for super windows heavy shops though.

Multiple ELB's came alive around that timeframe but our primary ELB has remained unable to re-register instances. Creating a new ELB as a test and trying to register new instances from the effected ASG has also failed.