undefined | Better HN

0 pointsfletom9y ago0 comments

what's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkmarks next to S3 on their service page.

they just now put up a box at the top saying "We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region."

increased error rates? really?

Amazon, everything is on fire. you are not fooling anyone

edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large

0 comments

29 comments · 13 top-level

idlewords9y ago· 5 in thread

@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."

jasoncrawford9y ago

Apparently this is not a joke:

“The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.”

https://twitter.com/awscloud/status/836656664635846656

1 more reply

mpetrovich9y ago

There are some real gems @Pinboard too: "Green checkmark = no lava in data center. Green checkmark with information icon = data center filling with lava https://status.aws.amazon.com"

paulddraper9y ago

FYI, in seriousness you can see the fabled red status icon here:

https://status.aws.amazon.com/images/status3.gif

It does exist, apparently.

1 more reply

tlrobinson9y ago

I thought this was a joke, but apparently not too far off:

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

https://twitter.com/awscloud/status/836656664635846656

scarlac9y ago

While that may be true, that's not the reason you're seeing green. You should have been seeing a broken image or a status page not finishing loading if that was an issue.

3 more replies

ckozlowski9y ago· 3 in thread

(Disclaimer: I work for AWS.)

The dashboard is not changing color due to the S3 issue. We're updating the banner in place of that.

Edit: Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

http://status.aws.amazon.com/

Ph4nt0m9y ago

For some reason, reading about "believe we understand root cause" made me think of: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

perlgeek9y ago

Maybe you could encourage your colleagues to host the status page outside of AWS?

3 more replies

danappelxx9y ago

Could you go more in depth? What does S3 have anything to do with it?

3 more replies

stretchwithme9y ago· 2 in thread

What we need are status pages that are driven by votes from verified customers, which could also serve to inform the provider about issues.

This would address issues that are only visible from the outside.

laughfactory9y ago

And system monitoring which isn't dependent on itself. Kind of a "duh" kind of thing...

ATsch9y ago

http://outage.report/ does this pretty much, except for the "verified customers" part.

artursapek9y ago· 2 in thread

I don't think they intentionally kept the checkmarks there. They probably just didn't update it as quickly as developers made a post on Hacker News (not surprising, they were probably investigating).

hobofan9y ago

After having seen multiple AWS outages/service disruptions, with nothing other than a green checkmark ever showing, I am now very confident that the checkmarks are hardcoded and there is no logic behind them.

1 more reply

fletomOP9y ago

when you start investigating an outage, that is exactly when you should change your checkmark to yellow if not red. if you're as big as AWS there should not be any more than a minute or two between when your service goes down and when you actually update your status page to show that.

1 more reply

j2kun9y ago· 1 in thread

> Amazon, everything is on fire. you are not fooling anyone

Fun story, when I was an intern at Amazon there was actually a warehouse fire. The result was a lot of manual database entry updating as products were determined to be destroyed or still fit for sale.

wildmusings9y ago

I'm curious about what happened to products that were no longer fit for sale, but still fit for use. Do you recall?

3 more replies

evtothedev9y ago· 1 in thread

The AWS Status page will lie to you: https://medium.com/@ev.dev.dev/the-aws-status-page-will-lie-...

fletomOP9y ago

I like how this post says "if you look at the AWS Status Page, this what you see". but you can't see the image. because S3 is down.

1 more reply

skywhopper9y ago· 1 in thread

I hear that their process for updating the status page involves S3.

fletomOP9y ago

it would appear that you are correct

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates." https://twitter.com/awscloud/status/836656664635846656

1 more reply

whafro9y ago· 1 in thread

"Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard."

Yep.

_ao7899y ago

I love it, like that fixes the problem! ..now fix the REAL problem

kevin_b_er9y ago

If this isn't good evidence that amazon downright lies on their status page and that no green checkmark should ever be considered trustworthy, I don't know what is.

twistedpair9y ago

Why is the status board hosted on AWS? Most providers host such pages on a 3rd party, specifically for this reason; correlated failure.

rinze9y ago

> edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large

So this is what centralization looks like.

knaik949y ago

So... https://twitter.com/awscloud/status/836656664635846656

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

gtrubetskoy9y ago

the non-green icon is probably hosted on s3 (i'm not trying to be funny)

j / k navigate · click thread line to collapse

0 comments

29 comments · 13 top-level

idlewords9y ago· 5 in thread

@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."

jasoncrawford9y ago

Apparently this is not a joke:

“The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.”

https://twitter.com/awscloud/status/836656664635846656

1 more reply

mpetrovich9y ago

There are some real gems @Pinboard too: "Green checkmark = no lava in data center. Green checkmark with information icon = data center filling with lava https://status.aws.amazon.com"

paulddraper9y ago

FYI, in seriousness you can see the fabled red status icon here:

https://status.aws.amazon.com/images/status3.gif

It does exist, apparently.

1 more reply

tlrobinson9y ago

I thought this was a joke, but apparently not too far off:

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

https://twitter.com/awscloud/status/836656664635846656

scarlac9y ago

While that may be true, that's not the reason you're seeing green. You should have been seeing a broken image or a status page not finishing loading if that was an issue.

3 more replies

ckozlowski9y ago· 3 in thread

(Disclaimer: I work for AWS.)

The dashboard is not changing color due to the S3 issue. We're updating the banner in place of that.

http://status.aws.amazon.com/

Ph4nt0m9y ago

perlgeek9y ago

Maybe you could encourage your colleagues to host the status page outside of AWS?

3 more replies

danappelxx9y ago

Could you go more in depth? What does S3 have anything to do with it?

3 more replies

stretchwithme9y ago· 2 in thread

What we need are status pages that are driven by votes from verified customers, which could also serve to inform the provider about issues.

This would address issues that are only visible from the outside.

laughfactory9y ago

And system monitoring which isn't dependent on itself. Kind of a "duh" kind of thing...

ATsch9y ago

http://outage.report/ does this pretty much, except for the "verified customers" part.

artursapek9y ago· 2 in thread

I don't think they intentionally kept the checkmarks there. They probably just didn't update it as quickly as developers made a post on Hacker News (not surprising, they were probably investigating).

hobofan9y ago

1 more reply

fletomOP9y ago

1 more reply

j2kun9y ago· 1 in thread

> Amazon, everything is on fire. you are not fooling anyone

wildmusings9y ago

I'm curious about what happened to products that were no longer fit for sale, but still fit for use. Do you recall?

3 more replies

evtothedev9y ago· 1 in thread

The AWS Status page will lie to you: https://medium.com/@ev.dev.dev/the-aws-status-page-will-lie-...

fletomOP9y ago

I like how this post says "if you look at the AWS Status Page, this what you see". but you can't see the image. because S3 is down.

1 more reply

skywhopper9y ago· 1 in thread

I hear that their process for updating the status page involves S3.

fletomOP9y ago

it would appear that you are correct

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates." https://twitter.com/awscloud/status/836656664635846656

1 more reply

whafro9y ago· 1 in thread

"Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard."

Yep.

_ao7899y ago

I love it, like that fixes the problem! ..now fix the REAL problem

kevin_b_er9y ago

If this isn't good evidence that amazon downright lies on their status page and that no green checkmark should ever be considered trustworthy, I don't know what is.

twistedpair9y ago

Why is the status board hosted on AWS? Most providers host such pages on a 3rd party, specifically for this reason; correlated failure.

rinze9y ago

> edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large

So this is what centralization looks like.

knaik949y ago

So... https://twitter.com/awscloud/status/836656664635846656

"The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates."

gtrubetskoy9y ago

the non-green icon is probably hosted on s3 (i'm not trying to be funny)

j / k navigate · click thread line to collapse