I've never implemented something like this. But my first thought is "how do you implement the testing system?"

I feel like once you had a robust testing system that can verify things work correctly in all the different network partition and other scenarios, and allowing rapid iteration of setting up those scenarios, the implementation would be comparatively easy.