"AWS Shield Advanced comes with DDoS cost protection to safeguard against scaling charges resulting from DDoS-related usage spikes on protected EC2, ELB, CloudFront, Global Accelerator, and Route 53 resources. If any of these protected resources scale up in response to a DDoS attack, you can request Shield Advanced service credits through your regular AWS Support channel."
Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. I've run SaaS where a customer forgot to send their key and their software would just keep retrying requests to the point that they were sending thousands of requests per second with no key. However, it's kinda crappy when it's happening through no fault of your own.
One doesn't even have to make unauthorized requests to run up a bill. One can simply make lots of spam requests to a real file in a bucket - run up the request total and bandwidth at the same time.
Who pays when the unauthorized requests contains truly random data, i.e. they point at an org and bucket that do not exist? Amazon does. Part of their cost of doing business is building the infrastructure that routes authorized requests correctly, and if a request is for invalid data, it's dropped. That should be par the course.
Large numbers of unauthorized requests can be effectively considered a DDOS on the service itself. That the attacker can add a note onto their attack that says "btw bill X customer for all this" and have that honored is ridiculous.
They cost a lot less than a multiple disk write (PUT), a disk read (GET).
They cost the same to AWS whether the target bucket belongs to me, Amazon themselves, or does not exist.
And in this case, the customer definitely didn't "screw things up".
Charging for unauthorized requests is just weird and customer-hostile. Like, how am I supposed to prevent that?
I get that a service provider might want to "punish" a customer who misconfigures their systems and DoSes themselves with e.g. no backoff on 403 error, but I find it hard to believe that that's the common case -- and how is typo in authentication supposed to be so different from typo in bucket name?
We really need some kind of regulation to ban predatory cloud pricing.
I had two old buckets left over from when I was using Elasticbeanstalk 5 years ago, and they had predictable names with my account ID in them. And as we heard in February, account IDs are easily leaked and not considered private.
Besides that AWS themselves recommend you to use your domain name as bucket name when using S3 static site hosting. Thankfully I've been using Cloudfront infront of randomized bucket names for a few years already.
So now I'm forced to increase the random string I use in all my bucket names from 8 to 16 characters. Every bucket is now a password for a malicious actor who wants to attack your finances.
Obviously all accounts should have budget alerts, but there is no budget ceiling feature.
Technically AWS does offer you all the tools to create a budget ceiling feature using budget alerts and lambdas that will shutdown the affected service. I just wish someone would do it because I don't have the time.
> Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
Abandoned GCP API gateway because of this...best as I can tell even with a rate limit the calls for the gateway get charged. I guess if it helps if the thing behind it is computationally expensive but still feels a bit silly
The bucket is the same name as the domain so I can imagine this could be exploited quietly very easily as PUTs are so expensive.
Literally a $5 VPS could cost someone $1k.
Yes, you would imagine, because without relevant WAF rules in place there would be no throttling.
Does AWS have default per-IP rate limits?
* PUT, COPY, POST, LIST requests are $0.005 per 1,000 requests
* GET, SELECT, and all other requests are $0.0004 per 1,000 requests
It sounds like these are getting charged at the PUT request pricing. And IIUC, those prices are set because successful write operations seek on multiple HDDs (physically moving the head) to commit the data to multiple machines, or S3 would be unable meet durability goals. Successful read operations are an order of magnitude cheaper because they likely move only one of several heads, sometimes none at all (because they're cacheable).
Authorization failures should never hit an HDD.
Other folks have mentioned that these auth failures are costing AWS something. Sure...but very very little. Far less than the cost of a successful GET, much less a successful PUT. They really shouldn't be charging at all for this, or if they do, it should be like $0.00001 per 1,000 requests (another order of magnitude cheaper than GETs). Not even that really; the actual cost for them to process this request is likely less than they're (over)charging for the bandwidth alone.
Essentially every S3 bucket, public or private, whose name is discovered can be DDoSed creating an insane bill...
That's a platform level security issue for AWS to resolve. We do not need another Bucketgate.
[1]: https://twitter.com/Lauramaywendel/status/178507498639629151...
[2]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
My personal suggestion is that you should never use S3 for public access. Public buckets are an open gate to your AWS bill. If you have private buckets the attack surface is reduced but even in this situation you cannot prevent billing attacks. Try to reduce the damage by activating Cost Anomaly Detection.
[1] https://blog.limbus-medtec.com/the-aws-s3-denial-of-wallet-a... [2] https://aws.amazon.com/aws-cost-management/aws-cost-anomaly-...
This should be entirely irrelevant, as those same failures were happening even before the bucket existed, they were just hitting a different failure path.
Yes for GCP and Azure, also for GPT...For Backblaze I am still investigating...
Two quick examples in different contexts:
https://learn.microsoft.com/en-us/answers/questions/1064419/...
https://community.openai.com/t/cost-for-failed-gpt-requests/...
Edit: It seems for Backblaze that would be a no, see below:
https://www.reddit.com/r/backblaze/comments/16o4bed/charge_b...
> Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
Any way to mitigate this on AWS by restricting access to it via VPC only?
If they actually charge for unauth'd requests that's absurd
My first thought is that can be abused to multiply the damage if you want to engage in some cost based denial of service.
About every other week AWS gets out of their way to make us remember that they are the worst cloud around.
For reference https://github.com/ZJONSSON/node-unzipper/issues/308
> Today I was hit by a surprise 500$ bill by using byte range request within zip files without an upper bound, but not consuming the whole stream. I tested for about 45 min at about 1gbps. My ISP meter says I downloaded 300gb which lines up with physics, but AWS says it was 6000gb.
Re the bill itself - You may be able to get the surprise bill waived by AWS with a support request, especially if you haven't had to do that before.
Re the metered billing - This isn't super clear anywhere I could find in the AWS docs, but I think from AWS's perspective your network data transfer out from S3 is the amount of data you requested + initiated, not what your app ultimately happened to consume from the stream. They really do not make this clear though, which might help your case.
FWIW, it sounds like AWS's implementation is behaving consistently about "range specifiers" with the RFC they link to in the S3 docs.
https://www.rfc-editor.org/rfc/rfc9110.html#section-14.1.1-4
Sidebar, but: If it works for your setup, I think putting a CloudFront distribution in front of the bucket and accessing that way would mitigate the damage by caching the whole object after the first request. I've used a setup like this for streaming very large files to many viewers before.
https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
500$ is not making a huge dent in the monthly bill for this. I just got a few laughs at my expense from colleagues..
I don't know how AWS meters, but, if your assumption is right, I can abuse this and generate a huge bill to anyone just by starting streams and not consuming then. I could generate 10s of gbps of "apparent" billable traffic with a puny VPS.
Edit: if they do that, they should make it possible to firewall buckets via ips so that they are only accessible via certain ips
The problem is the cloud priorities availability over cost savings. We used to run our own infrastructure, that would just fall over if it got too many requests. This infrastructure never falls over; it just bills more. I don't know the right way to solve this, other than don't stick your S3 bucket on the public internet.
[1] https://github.com/automaticit-anthonyyen1/s3-bucket-name-ge...
1. If you are using buckets for static website hosting, amazon's own docs require you to use the website name as bucket name.
2. If you are using pre-signed urls, they expose the bucket name.
3. Bucket names must be between 3 (min) and 63 (max) characters long and have various limitations. So, it's not too hard to guess bucket names and run millions of requests without even having an account.
${AWS::AccountId}
https://medium.com/@TalBeerySec/a-short-note-on-aws-key-id-f...
The question is more about how long AWS are going to take to fix this issue and how many DDoS bills will they forgive.
Aaaah! CloudTrail data events are priced in such a way that each S3 request gets logged--at a cost of 20x that of the S3 request itself. I really hope OP got THAT part of the bill waived as well.
not viable for a public bucket, but seems like it'd do the trick on private buckets.
edit: hahaha, i'm wrong https://docs.aws.amazon.com/AmazonS3/latest/userguide/Reques...
What's next? An article that explains why the bill stacks up when you log all denied fw packets?
If there are limits which are surpassed, S3 charges may be incurred.
