Failure Injection on Kubernetes with SMI and Linkerd (opens in new tab)

You mean like https://www.gremlin.com/ ?

One of the founders of Gremlin is an Engineer that worked in Netflix and probably worked on Chaos Monkey as well :)

These kinds of errors are going to happen in production, whether you inject them or let them occur naturally. Any release process that doesn't go perfectly with a drain / rebalance / start new version / rebalance per (backend, proxy) combo is going to have a timeout or broken connection between the proxy and the backend as it restarts. Should you return 502 to your users when that happens? Nope, just retry on a different backend. This lets you test that.

That's right, and there are some interesting trade-offs. Having failure injection built-in is convenient but running a separate service gives you full control over the error responses. This can be useful if you want to simulate responses with error bodies, for example.

It will be helpful for testing your retries, and getting insights into your client's behaviour in the event of service failures.