AWS Experiencing Intermittent DNS Resolution Errors (opens in new tab)

"We want to give you further information regarding the occasional DNS resolution errors. The AWS DNS servers are currently under a DDoS attack. Our DDoS mitigations are absorbing the vast majority of this traffic, but these mitigations are also flagging some legitimate customer queries at this time. We are actively working on additional mitigations, as well as tracking down the source of the attack to shut it down. Amazon S3 customers experiencing impact from this event can update the configuration of their clients accessing S3 to specify the specific region that their bucket is in when making requests to mitigate impact. For example, instead of "mybucket.s3.amazonaws.com" a customer would instead specify "mybucket.s3.us-west-2.amazonaws.com" for their bucket in the US-WEST-2 region. If you are using the AWS SDK, you can specify the region as part of the configuration of the Amazon S3 client to make sure your requests use this region-specific endpoint name.

The DNS resolution issues are also intermittently affecting other AWS Service endpoints like ELB, RDS, and EC2 that require public DNS resolution. We are actively working on this issue and will update you as soon as the issue is resolved on our end, however at this moment I won’t be able to provide an ETA. I am keeping this case in Pending Amazon Action, will update you as soon as I get further information on the resolution of this issue."

https://www.reddit.com/r/aws/comments/dlnl28/route53_is_fail...

Nothing listed on AWS Personal Health Dashboard related to this incident. Also nothing abnormal listed on the AWS status page under S3, which is seemingly having DNS resolution issues for buckets with the `[bucket_name].s3.amazonaws.com` pattern.

Switching to googleDNS 8.8.8.8 or 8.8.4.4 bypasses the affected DNS servers and will resolve the simple URLs (no region in url). Still really surprised this isn't making bigger news yet.

My company hasn't been able to deploy any updates to our ElasticBeanstalk application for about 7 hours due to this. Luckily there's nothing urgent we need to deploy, but this makes me extremely nervous about using AWS going forward. If we experienced an outage that required a rollback or forward fix we would be totally hosed.

And of course Route 53 still has a green checkmark.

AWS is not really "Too Big To Fail".

It's more like, "Too Big, Will Fail".

The internet is about to halt. SQS & SNS are not resolving from many parts of the world.

This is affecting my websites that use cloudfront, for which Amazon apparently uses their own Route 53 for DNS (nslookup -query=ns cloudfront.net). Since the only way to call remote assets from cloudfront as a website asset is to use: xyz.cloudfront.net or a CNAME such as mysites-cdn.com that maps to xyz.cloudfront.net it seems there is no way to use cloudfront without the Route53 lookup.

If route53 is down and that is required to use cloudfront how is this not affecting more people? I have had about dozen customers complain today.

In our case the issue is affecting DNS resolution of 'only' S3 related hostnames (my-bucket.s3.amazonaws.com)

Is Route53 under attack or a customer on Route53?

By all appearances the Route53 DDOS mitigation strategy is massive scale and distribution. This includes distributing customers and their NS records across infrastructure AND TLDs. I would have thought a blanket attack against Route53 impractical..

Maybe a stupid question, but what to do when eu-central-1.signin.aws.amazon.com is down?

If you got paged and are currently dealing with this, one thing you can do is set better defaults for AWS_REGION. i.e update the configuration of your client that is accessing the AWS resource to specify the resource's region

I’ve noticed CloudWatch Dashboards malfunctioning today (not showing any data).

We are facing this right now. Trying to push new pages to Cloudfront.

I was getting some weird notifications about "kms: server misbehaving". No production impact so far, fingers crossed.

Why does the AWS status say the issue is resolved when it obviously isn't? S3 is still down in parts of US

YouTube is having issues as well (via the Android app), does YouTube run partially on AWS?

Dont worry, the internet was designed to circumvent nuclear attacks like these

Is this the cache poisoning ddos posted earlier?

AWS has wasted so many hours of my life troubleshooting their “throw shit at the wall and see what sticks” services.

I don’t know why developers put up, even push for, their garbage services.

China