undefined | Better HN

0 pointschrisan9y ago0 comments

Down for us as well. We have cloudfront in front of some of our s3 buckets and it is responding with

    CloudFront is currently experiencing problems with requesting objects from Amazon S3.

Can I also say I am constantly disappointed by AWS's status page: https://status.aws.amazon.com/ it seems whenever there is an issue this takes a while to update. Sometimes all you see is a green checkmark with a tiny icon saying a note about some issue. Why not make it orange or something. Surely they must have some kind of external monitor on these things that could be integrated here?

edit: Since posting my comment they added a banner of

"Increased Error Rates

We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region."

However S3 still shows green and "Service is operating normally"

0 comments

2 comments · 1 top-level

gtsteve9y ago· 1 in thread

To mitigate the effect of S3 going down I use cross-region replication to replicate objects to another S3 region. If S3 went down in my primary region I could update the app config to write back to the backup region and update the CDN configuration to use the backup region as an origin.

I did that out of paranoia but it turns out this could happen to us. Does that sound like a sensible approach?

Fortunately all my company's stuff is in eu-west-1 which still seems to be fine.

kaishiro9y ago

I'm certainly not an expert here, but just to make you feel good (if nothing else), this is exactly what we did this morning (Melbourne time). Woke up to a bunch of flailing Lambda funcs on us-east-1. Luckily we're using Apex so cross deploying them to Singapore took all of 30 seconds. We were concerned about API Gateway since it was also sitting in us-east-1 but ended up not being an issue. Realized that redeploying S3 and Lambda across regions can be done in practically no time, but we would have been in trouble had we needed to replicated our APIs in another region. Going to start exploring spec'ing up our existing gateways in swagger to help with this.

j / k navigate · click thread line to collapse