I'm shocked that an automatic canary rollout wasn't an action item. Pushing anything out globally is a guaranteed failure again in the future.

Even if you want this data to be very fresh you can probably afford to do something like:

1. Push out data to a single location or some subset of servers.

2. Confirm that the data is loaded.

3. Wait to observe any issues. (Even a minute is probably enough to catch the most severe issues.)

4. Roll out globally.

Presumably optimal rollout speed entails something like or as close to ”push it everywhere all at once and activate immediately” that you can get — that’s fine if you want to risk short downtime rather than delays in rollout, what I don’t understand is why the nodes don’t have any independent verification and rollback mechanism. I might be underestimating the complexity but it really doesn’t sound much more involved than a process launching another process, concluding that it crashed and restarting it with different parameters.