This doesn't make sense at all. Amazon should let us if monthly bill > X send me a priority email and phone call. Why do they hide behind these dark patterns? I thought they were better than that.
The reward for focusing on this before-hand is much lower than just writing a check for $5k to this person and then fixing later (lot of $5k checks from amazon today. Wheres mine?)
Spot Instances aren't subject to AWS Billing Alerts? Is this common knowledge?
We caught and corrected it quickly, but we still don't know how the keys leaked out - we have chalked it up to lower security practices since it's not a production account and is shared by more people (e.g. no 2-factor on it). We started to investigate, but then Heartbleed happened.
I wish there were more mechanism in AWS to prevent bills from mounting up, but the basic billing alarms worked in this case. I can't imagine how or why spot instances would be excluded from alerts, their cost certainly is included in the estimates that alerts are based on.
Coincidentally, the incident also occurred around the same time (April 1-2). We were hit with $13,000 worth of EC2 usage before we shut them down and changed our AWS key... We reported to Amazon, and they are working on a refund.
A $5,000 AWS instance would mine about $1 worth of bitcoin and would not be worth the time logging into someones account.
