Fallacies of distributed systems (opens in new tab)

- We used gRPC with no compression before, and we tune an acceptable compression level to achieve a balance of performance and data volume.

- Improve our SQL optimizer to consider the network cost more accurately and data placement rule to balance data more logically to reduce data transfer crossing AZ.

- As a cloud service, we will also expose the cost transparently to our customers like MongoDB Atlas does. :-)

We didn't pay attention to the cost of crossing AZ before, just thought this might be cheap. Of course, we were wrong :sob:

Each failure scenario must be gracefully handled in each subpart to take into account all possible impact on the whole system.

You'll always get a random crap you never heard of destroying an assumption you made in your own crap.

I remember an instance where a national mobile phone provider in France got a single db outage but during a large sms day (think new year), so the failover happened but totally overloaded the new servers, and it created a chain reaction to other providers to a point you couldnt read your email for a day or two because of inability to get smses. Why is that random db in a provider I dont even use impacting my google account ? Because the system is distributed.

e.g. let's take the example of an init system starting up all processes. Now what happens if a if one of the processes crashes and gets restarted by a processmanager? Now the order already changed, and e.g. a former process which relied on the restarted one might work based on outdated data. Similar things can also happen on other layers - e.g. one of the services in a dependency chain might disappear and reappear.

Another example is a developer/administrator manually changing the config of a certain service and restarting it to take effect - that could also trigger dependency problems.

Now those are absolutely solvable - either by making sure all services operate gracefully with any startup order or by other mitigations (e.g. "always reboot the full box"). But like everything else in the list, it still is a problem that is observed in very often in distributed systems.

Much better to lose a node immediately than to have it fail gradually.