It's often an unpopular opinion around here, but this is why I prefer simple hosted databases with limited query flexibility for high volume and high availability services (Firestore, DynamoDB, etc.). It's harder to be surprised by expensive queries, and you won't have to fiddle with failovers, auto scaling, caching, etc. Design your system around their constraints and it will have predictable performance and can more easily scale under unexpected load.
I hope all their hardware crashes are also scheduled when there will be no impact... This seems a bit backwards - unless you constantly exercise the instantaneous failover, how do you know it works?
Edit: Actually it's worse - if you don't test the instant failover under a full load, how do you know it's still instant then?
2. Loadtesting? Accurate end-to-end loadtests are painful to bootstrap from nothing and requires an environment that can be expensive, worth their weight in gold depending on how critical downtime is or if everything in your platform is pulling from the same resource pool(ie appliance based deployment).
If they are a bank, this isn't the end of the world. I've had online banking outages at "normal" banks. It is still a bad thing, but there are other ways I can get my money, like going to a branch.
On the other hand, if Coinbase is like a brokerage, this is really bad. And let's face it, most use of crypto is for investment and speculation purposes. For trades to fail for half an hour is really bad. If they are running this thing like a startup on MongoDB (seriously?) I don't see how anyone who puts their money in can have any confidence of getting it back out.
Do you base this on recent info (latest versions w/ Jepsen tests)? If so, what specifically makes Mongo a "startup" db?
I thought this was interesting. I think that caches can be so dangerous in an incident - suddenly operations that are almost always constant time are executing in a much different complexity, and worst is that this tends to happen when you get backups (since old, uncached data is suddenly pushing recent data out).
I think chaos engineering may be a good solution here, in lieu of better architectures - see what happens when you clear your cache every once in a while, how much your load changes, how your systems scale to deal with it.
Based on my experience with largish data in mongo, my guess would be that the database size was >> than ram and, due at least in part to mongo's design, when the master failed over the memory state of the new master didn't have the working set present in RAM. This lead to a huge inrush of disk IO resulting in all sorts of bad until the RAM state sorted itself out.
Haters gonna hate, hate, hate
No you don't.
- If you did, you'd hire a DBA team and they would be familiar with the various jobs in your environment. But first your founders would have to have respect for Operations, which will take a dozen more major outages.
The other major Coinbase outages have also been database-related, namely missing indexes.
- If you did, you wouldn't be doing major database (or other production) changes at 3 pm in the afternoon.
So let's cut to the chase. You prioritize features over Operations, and as a result guinea-pig your users. Just like any other SF startup. So just admit that to your end-users.
Coinbase isn't dedicated to one timezone. They fully support tons of countries and have customers all over the place.
Things like this have to happen at some point, and there are benefits to doing stuff like this during "work" hours (like having all of your staff online and available)
What timezone are their developers in? That’s the important question in this situation.
Devs deploying changes that affect customers at the end of the devs’ workday are reckless.
That sounds like a benefit to Coinbase and not to any of their customers.
Coinbase had a number of issues when cryptocurrencies really exploded in 2017, and at that time I felt more willing to give them the benefit of the doubt because the landscape of cryptocurrencies had shifted so dramatically and I could empathize with their struggles to keep up. Two years later, there aren't any excuses anymore in my mind. As the parent comment says -- it's just not a priority.
I've happily moved off the platform to other options which have given me no trouble whatsoever.
Wow, I'm not from any "SF startup" but I find that side jab quite cunning.
