All distributed decentralized systems are brittle. The only people who don't think this are people who haven't run them at scale.

Also, "self-healing" isn't really one thing. There are hundreds of different problems that can take out such a cluster, and every single one of them needs its own "self-healing" mechanism. These systems are literally the most complicated kinds of systems.

I haven't been responsible for babysitting consul, but I have been responsible for etcd for years and if consul's problem is anything like etcd's it's because members have identity and if one of them goes toes up then etcd will wait forever for the snowflake to come back to life, and if that's not how the underlying infra is configured, that's very very bad. Mix it mTLS into this story and it gets worse

I stayed away from the so-called "stacked" control plane of etcd inside kubernetes because it can make a tiny fire into a sharkfirenado but recently I've heard discussions of k3s (which uses dqlite) managing the etcd members and then "formal" kubernetes managing the workloads pointed at that k3s-stacked-etcd but I haven't tried it yet in order to know how theory and practice differ

I've worked with Vault and clusters, not Consul specifically but generally there's a healthy cluster state until something happens putting the cluster into an inconsistent state.

Generally there's a master node or multiple nodes in agreement. If the cluster cannot agree on it's current state the entire system may run multiple versions or be completely unavailable or provide inconsistent response bringing down other systems that rely on it.

Inspection itself is hampered by elections or syncing state or other process/race related/caching/ddossing itself or other services.

Consul is a lot more than just a database cluster, and that may be part of its problem.