AWS status updates not working due to S3 (opens in new tab)

Writing a B+ tree from memory and making sure your infrastructure isn't doing something stupid are fundamentally different skills. One requires that you regurgitate the contents of a text book on a white board, the other that you can engineer a solution. I wish them well on an interview set up for hiring the former; I try to hire the later.

> cannot be made by people who had the smarts to write b+ trees in 15 minutes...

Writing B+ trees at the drop of a hat is probably more a signal of memorization and recency of taking a data structures class than smarts, particularly the smarts necessary to develop and maintain robust distributed infrastructure.

1. The number of people at Amazon who need to be able to write a b+ tree is negligible.

2. The number of people at Amazon who need to be able to choose a search/sort/etc. algorithm or data structure and understand why one is more appropriate than another for a given use case is much higher.

3. The number of people at Amazon who need to demonstrate common sense is very high. This skill is much more closely related to #2 than #1.

I doubt whether tech companies have data to backup for this type of interview, something like the correlation between people who do well in whiteboard binary inversion interview and people who do well in real world jobs. Or they just do it because that's the way Google does it.

>Such mistakes cannot be made by people who care.

That's a very naive assertion. Humans make mistakes, they always have and they always will, no matter how smart they are and how much they care. That's why pilots have checklists that they go through before they're even allowed to leave the gate.

Because bureaucracy, siloization, Tragedy of the Commons and likely a massive infrastructure with tons of technical debt and inability to make major changes except by gradual incrementalism, oft too late. Infrastructure needs active SimianArmy-style breakage finding without "sacred cows" but with less than 100% uptime across all services to ferret-out outage edge-cases.

What if I told you you could make a red dot without hosting an image anywhere?

Seems like another commenter beat me to the punch, but to me the obvious answer would seem to be not hosting images at all as they could create a red circle in css.

Or as others noted reverse the logic so that it shows red icons by default but as long as the services are working then it replaces that with a green icon. And when those external services are down it would go back to a red icon.

It is...it's literally the reason products like statuspage.io exist, because if your status page has any dependencies on the services for which is provides statuses, then it's not really a useful status page.

The status page should work with just IPv4 or IPv6, BGP and round-robin on a bunch of location-diverse, simple, real metal, web-boxes that only serve status.

I would hate to imagine drones dropping out of the sky if S3 went down in the future.

Meanwhile, on InfoWars "S3: NWO AI mothership ready to hijack everything" ;)

Cracker! ROTFL

> 1. ensure it does not depend on your infra (if your api server goes down - it should not take down your status api with it)

Many people forget DNS in the equation.

If it's on a subdomain of your regular site, it will go down in case the domain is accidentally/maliciously transferred or legal authorities seize/block it (we're seeing the extremely long arm of the US law enforcement with Mr. Dotcom, as well as Erdogan and other dictators or the Chinese firewall).

If it's on a different domain that's on the same DNS hoster (e.g. Amazon's Route 36, or for that matter your own hoster!) you're screwed if the DNS fails.

If it's via the same registrar, you're screwed if someone obtains access to your registrar account (this once again includes law enforcement).

Obviously this also holds true for the TLD itself - e.g. imagine Verisign (holding .com and .net) has problems, you want a .info, for example.

Conclusion: different datacenter/provider for the HTTP server part, different DNS provider(s), different TLD. For the datacenter and DNS provider level you can use high-availability (multiple different NS entries, multiple different servers), this can also protect from legal overreach.

Also, your status page may have a negligible load as long as your service is operating fine, but people tend to go to status pages and manically press Cmd+R until there's a green light - so best use nginx/lighttpd with static pages and minimal assets only.

If you're running HTTPS on your main site and you do choose to name it "status.mydomain.com", also deploy HTTPS on your status page - else people visiting status.mydomain.com may transmit session cookies in cleartext in case you forgot the SECURE flag or the client does not honor this (for whatever reason).

Oh, and do buy a separate HTTPS cert instead of using your usual wildcard cert or your primary cert with the status page as SAN, so your status page stays up when your primary cert expires...

Not sure about 2 - What are your arguments for this?

If the status page relies on getting updated information from the service, it may not even notice when the whole thing just crashes and goes down in flames. Attempting to do some predefined calls to the service to evaluate whether it is working correctly appears like a better solution?

Assuming that it actually makes business sense to do so. There are certainly cases where you can make a perfectly rational business decision to depend on someone else's services and you're OK with your uptime not being any better than their uptime.

> It's a similar mistake of making DNS a dependency for monitoring/control infrastructure when DNS is down.

This one is relatively easy to "fix" at least, it's nothing having multiple DNS providers for public records can't handle as well as ensuring redundancy for your internal DNS services. Bonus points if you run your own recursive resolver so you aren't dependent on some other party not screwing up somehow.

That's good practice, sure, but their problem was even more fundamental than that. Their status page was dependent on the service it was reporting on being up. That fails the most basic requirement of a status page.

yup - https://twitter.com/awscloud/status/836662601090134017

One explanation might be that they use an internal tool to update the status page definitions, and parts of that tool are hosted on S3. Or that the status definitions themselves are hosted on S3 (and then read and transformed into the HTML page everyone sees)