We got ourselves into a bind because of Nomad. We outlined a bunch of it here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchest...

The short version is that "using Nomad and Consul for the type of global workloads we run is not a good choice". I do not believe we'd have the same problems with Nomad + Consul in a single region. But running a single, global cluster of each of these is suboptimal.

The second problem was using some Consul features that forced us to keep it single region. What we actually need is a global view of a single service. Federated Consul doesn't quite give us that. Earlier versions of our infrastructure were using a bunch of Consul watches to update local state, so we couldn't really federate.

Some of this I'd do very differently if we rewound. But we were also building an idea with no actual users. Nomad and Consul gave us a nice platform to experiment on. We just outgrew the "prototype" as we learned what people actually wanted from us.

Hopefully they'll blog about it at some point. At this point I might pay just for their blog posts, even though I'm not using fly (yet?) because it's certainly a more cost-effective way to learn about big architectural mistakes than the way they're going about it, which involves actually paying the sunk costs.