> I’m writing this short post to chronicle the events that lead to this dreadful accident along with a few important lessons to learn from this fiasco.
> Before we begin, a quick overview of my fist. I was wearing a knuckle dusters at the time, but this could happen to anyone that has a hand. I decided to start placing my fist into my own face multiple times, with the most force I could.
Not this one.
1. It isn’t a rite of passage to delete a production database, it’s an extreme accident / fuck up. Stop normalising this. The guy literally deleted the machine without even thinking about the impact.
2. There was no backup. At all. Not even one. Not even one from a week ago. I can’t have any sympathy for someone hosting a production database without any backups running at all. A backup didn’t fail - they literally had purposefully not set up a backup.
3. Using a cloud provider and hosting your entire infrastructure on one virtual machine including your critical persistence layer with no replication. This is so stupid I can’t even begin to feel sympathy.
There’s nothing to be learned from this article other than the fact the author makes it a habit to implement bad practices.
Using an ASG to manage a production database is a tale from the land of What The Everloving Fuck Were You Thinking, as you note, the author makes a habit of bad practices.
It's at least instructive to note that there is only one canonical good practice when working with AWS:
Always assume you are handling a footgun until you have proven you aren't.
While they have improved the AWS::RDS resource in that there are fewer ways in which small configuration changes will drop your database, there are still big red banners in the documentation.[1]
To do it, you must specify a policy to say, "please don't nuke my prod database."
Because AWS will always default to being a footgun.
[1]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...