netsectoday

After having my domain hijacked today I'd like to share an incident postmortem just in case you find yourself in the same situation (swearing at your terminal output at 10am on Friday morning when your website is down).

9:58am I can no longer deploy to production

10:15am Finished troubleshooting all services, no problems identified

10:16am nslookup resolves to some random IP address instead of my prod server (WTF!!!!!!)

10:20am Log into registrar and find out they replaced my custom DNS servers with their own and added records to serve a "Parked free courtesy of GoDaddy" page with ads and a button that says "Get This Domain"

10:30am Changed my domain on the registrar website back to my custom DNS servers

10:32am Changed my password on the registrar website

10:38am Got told by GoDaddy support they didn't have anything to do with this and it was my fault it happened (f-me, right?)

11:55am DNS records across the internet are still jacked

12:00pm Manually blow out the cache on cloudflare for my domain

Postmortem Suggestions:

* If your website goes down; don't blow 15+ minutes troubleshooting your app services before checking DNS

* Enable 2fa with your registrar (even though there was no alert for us)

* Set up an alert for when your domain resolves to a different IP address (make a script and host it elsewhere or pay for a service)

* Don't trust your registrar!!!!

* Take a screenshot of your registrar settings and DNS settings right now so you have a record when they disappear

* Get access to your registrar account ASAP after the attack and change your DNS records back using the screenshots you just took

* Manually purge the cache of major DNS providers (for your domain) to allow your DNS records to propagate: https://cloudflare-dns.com/purge-cache/

My analogy is that we are designing bowling alleys. Setting up pins and creating the lanes with the hopes that people will bowl correctly. When a beginner comes along we inflate the bumpers to help them down the lane.

Up until today this has been my understanding. But now, after releasing a specific feature to production, I realize the "beginners" aren't toddlers that need a little help getting the ball down the lane... they are actually 900lb gorillas walking in to my bowling alley, finding the stash of 22 pound bowling balls, facing with their back to the pins, and launching them wildly like the first pitch at the world series.

What's your analogy to software engineering?