AWS faulty UX cost our business $7k and they refused to do anything about it (opens in new tab)

In their gif, the big alert right at the top, in the very first bullet point, says it will make a copy of the object. They intentionally scrolled down quickly in the gif to hide this.

Is the argument that this should have been a pop-up that required you to click “ok” instead?

"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."

The extra dramatics in this article turn me off.

Its essentially saying I'm busy and negligent. Now I'm mad big company didn't come to my rescue.

> scaling our business from 3 to 10 and then 100 people, raising money, editing our film, and planning our release. Oh, and a virus shut down the world, I moved two times, and our business went completely digital.

> and amid 18 hour workdays,

The quantified communication with company provides no details to what was said.

Maybe op was right, maybe they weren't but this sounds like whining without any evidence.

> After this experience, I cannot honestly recommend using AWS unless you have a professional sysadmin operate it for you, even if you think you know what you’re doing.

Maybe you should step back and think why people like us put our blood and sweat and it still takes years to learn how to administer infrastructure properly. Stop whining and learn to pay for skills.

I've never worked with a tool that made me this scared to use it. I mean it.

One wrong click and you can blow your budget, crash you entire system, the works. Over the years I've grown used to it but the feeling of uneasiness will never leave me. I get it, AWS is very powerful but it does not give me the comfy feelings of using a product like Render or DigitalOcean, where I pay for what I need and that's the end of it.

Is S3 marketed towards nontechnical folk like this? The other comments in this thread do a good enough job at dispelling any need for sympathy to the affected party, but my question is how they ended up inside the AWS console in the first place? It looks like the guy had just enough knowledge to shoot himself in the foot.

The little animation shows some alert thrown by AWS when the author goes through their process, but it goes by too fast for me to make out the details. I'm not going to watch it 100 times to get it all, but I do see phrases about copies and folder objects that might be relevant to this issue. Perhaps warning about the limits of the action being taken? I can't tell.

We had something similar happen with lifecycle rules, moving an entire bucket to glacier: end result was > $200k and AWS weren’t especially helpful on our attempts to refund us.

We were unaware that changes via a rule still count as an api hit per object, despite all happening in the backend. I’m sure we can’t be the only people hit by this.

I use AWS daily. It takes a few days to figure out those warnings are mostly there because some customer found out the hard way what they mean. You ignore them at your own peril.

And seriously:

> This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule.

I hate the AWS Console UI for a bunch of reasons, but short of making you type that sentence before you proceed, I'm not sure what AWS can do here.

Amazon makes a bajillion dollars a second. They could have made him happy and an evangelist for their customer service at zero impact to them. They just wanted to be dicks about it.

And folks, this is why you hire system administrators. We understand how stuff runs, is stored, backup procedures, cost expenditures and savings, and more.

We also know dev work, but it's not our specialty. We also know how to manage projects but we're not project managers. We also know how to handle billing, but we're not accountants.

... But our job encompasses all of them. And we reach out when we don't know. Our job is to make the trains run on schedule, using the average amount of fuel, with nobody stressing, and the customers happy. You get rid of us.. and... $7k of overage is nothing.

A lot of people have this mental model that The Cloud is Magic™.

They say things like this with a straight face:

- The cloud is always cheaper than on-prem!

- The cloud has no downtime, unlike on-prem!

Recently, I came across a large government department that had moved a reasonable volume of stuff into Azure. Think 7-figure annual bills.

They didn't like that they were overspending, so they complained to Microsoft, essentially demanding that they "make good" on their promise to make the cloud cheaper by cutting the cost on resources. This would have required an 80% discount, maybe higher. It's an absurd thing to even ask, but ask they did. Microsoft said no.

Meanwhile, I looked at their tenant, and the stuff that goes on there is amateur hour: Daily uncompressed database backups kept forever in premium zone redundant storage. Dozens of misconfigurations like that just burning money.

The worst part of it is that the "Azure Advisor" tab on the portal literally calls this kind of stuff out, recommending fixes worth hundreds of thousands of dollars a year, many with a one button "Quick Fix!" that you can press for an instant discount.

Nope. No. They're not going to bother reading things, or checking the Cost Management portal every now and then. They'd rather complain to their Microsoft rep every few months in a huffy tone.

"You still haven't cut my costs! Why haven't you!? I thought that's what you people did!"

They're busy people and don't have the time to waste cutting half a million dollars of spend here or there with arduous tasks like button pressing.

OP here. Thanks to some of the helpful comments, it seems that the creation of a copy when I chose "Edit storage class" may have been the culprit.

I didn't suspect that because in all my correspondence with Amazon, they always said the issue was incomplete multipart uploads.

So I assumed, based on the dialog box called "Edit storage class", that the net result of this action would be the same files but with a different storage class. This led me to read the warning as "We will create a copy of your objects and delete the old one, so please be aware you will be charged for the cost of the copy" rather than "We will create a copy of your objects and two versions of them will exist forever."

So it does seem like Amazon tried to warn people. But why on earth would a dialog box labeled "Edit storage class" not do anything of the sort? Are there actual legitimate reasons someone would want to have two versions of their objects sitting around?

Say what you will about my billing awareness (and you are saying it. Feeling the hate, all, thanks) but this is not good UX.

If there's any legitimate reason to duplicate objects in the name of changing the storage class, I'd love to hear it.

I dunno... it might be bad UX. I have definitely been annoyed with limitations to bulk edit abilities from S3 console before.

But I've been in this business for a while. I'm a programmer rather than an ops person, but i still sometimes make changes like that. I am responsible for some S3 buckets of stuff whose cost is significant for our budget.

I would never make a change in any software, where a failure for the change to take would cost me a noticeable part of budget, and not do even a cursory check to see if the change took. I mean, I write software, I know how it goes.

Who does that without ever checking to see if it worked? I don't do anything that matters without double-checking to see if it worked like I expected, at least if I'm doing it for the first (or second, or third) time, whether via a console or a script. Cause if you work with computers, you know they don't do what you expected all the damn time.

You're making a new kind of change you have never done before? You try it with a few objects before trying it on your whole bucket. You double-check if it did what you expected. Then you do it to everything. Then you still double-check to make sure it did it as expected to everything. (Even if you "read the manual" as the OP says, it's not about that at all in fact). That's how engineering works. If you don't have time to do it carefully, then sure, your chances of making mistakes or misunderstandings goes up a lot, why would you expect someone else to be responsible for your mistakes from not being careful?

I don't know, maybe there's a legitimate UX issue here, but the author also said it only took them an hour of reading up on Glacier to resolve. That seems a reasonable amount of effort to have put upfront.

It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges.

I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day. I don't expect them to step in like that for a problem sustained over the course of months.

I've never used AWS before so you can consider this a "fresh perspective". In defense of AWS, the warning starts with "This action creates a copy of the object". In defense of the author, "Edit storage class" is a pretty misleading name for the operation (how about "Copy with new storage class?"), and perhaps these "warnings" show up under other more routine operations frequently enough --- and unimportantly-enough to cause warning-fatigue.

However, given the nature of cloud services in general (they have a strong $$$ motivation to nickel-and-dime you for every little thing they can) and especially AWS' reputation for doing that, I likely would not have made this mistake. Then again, I'm also someone who carefully reads all available information --- especially whenever it's something I'm paying for.

Ignorance isn't bliss I guess and if you have the budget to hire 100 people, you have the budget to hire someone who has a clue about the cloud.

While the cloud is just someone else's hardware, you still pay for it like it's your own.

Okay that sucks and I understand where the author is coming from. There are definitely some rough edges in the AWS interface. I do believe as a large corporation Amazon does have some responsibility to waive some of the charge. In this case the intent was not to store things in S3 and just leave it there (especially if there was no access).

Amazon has traditionally been good about waiving charges for account compromises and things along that nature. However you do have to keep up on it since they bill you every month. If you're running a company part of the responsibility is at least checking in on your financials every so often. Right when you see a charge for $700+ when you expect $50 is a red flag. At that point if he had contacted Amazon I'd be surprised they didn't waive the charge.

I'd be curious if Amazon was willing to waive any part of the charges at all.

My perception is that with AWS, instead of managing a physical data center, you have to manage a virtual one. You still need dedicated technical staff.

A small business should use a backup service like Backblaze. AWS is what you use to _build_ a backup service.

I'm the co-founder of a startup named Vantage that is an alternative to the AWS console with a focus on developer experience and cost transparency and this is exactly the sort of thing we try and help people with.

We're sorry that this happened and hear about it a lot. I'd highly recommend folks sign up for Vantage if for no other reason than to get our weekly cost reports which would have caught this sooner: https://www.vantage.sh/

I'm personally curious how a 100 employee company gets almost bankrupted by $7k?

This article should have been written in a less bombastic fashion, I felt at the end like it was just an advertisement for a film.

Turn on anomaly detection in your billing, it will pay for itself within 3 months. Everyone screws up something in AWS eventually.

We had a run away EFS volume that burnt through $300 in a day, luckily was caught by an alarm. Thats basically $8700 saved if accounting had caught it first.

The interface is awful. If he had said something after the first bill, I'd be 100% on his side.

But he waited 9 months and kept paying. That's just too late.

Ah, reminds me of the time I ran through $15k worth of compute in all of 3 weeks trying to do biological modeling. What was strange was that we were being charged for higher bandwidth even though it was not said to be any more for that same instance even if the usage stayed the same. We switched to Digital Ocean after that and that problem went away and had an all you can eat set of arguably better systems (32 CPUs vs 16) for $640 a month per device. So yeah, really do your homework, keep up with the notifications, and realize that they LOVE people like you! Why do you think cloud companies are so rich? Every startup idea gets put on there and either forgotten or blows up and ends up spending more in services. So yes, keep this in mind moving forward and dont listen to people who swear by AWS for x y of a feature because chances are they dont actually know what's going on behind the hood but have had to he trained it or heard about it in tech interviews if they are a cs person.

The minor lesson here: be careful operating heavy machinery with no cost limits -- or hire someone who knows what they're doing to do it for you.

The major lesson: audit your expenses every month (or hire someone to do it for you). The AWS UX cost them $800. Faulty business administration cost them $6200 on top of that. It could have just as easily been $MAX_CREDIT_LIMIT.

The fact that you cannot set a hard limit or preload funds for a bill cycle is key to their business

For all the people arguing that the presence of a box with some text spewing warnings in technicalese constitutes "UX", and that the assertion that bad UX costs money is false due to the presence of said box:

I don't think you understand what UX means.

I honestly can't even imagine not checking my AWS bill for 8 months.

> I was the most technical person on our executive team, and amid 18 hour workdays

This shows how incompetent this guy really is.

There’s a great book called Extreme Ownership. You really should read it.

just do go to AWS, it's easy