undefined | Better HN

0 pointshughrr4y ago0 comments

As a fairly regular consul cluster admin for the last 6 years or so, but not on that scale i can safely say that you generally have no idea if rolling back will work. I’ve experienced everything up to complete cluster collapses before. I spent an entire night blasting and reseeding a 200 node cluster once after a well tested forward migration went into a leadership battle it never resolved. Even if you test it before that’s no guarantee it’ll be alright on the night.

Quite frankly relying on consul scares the shit out of me. There are so few guarantees and so many pitfalls and traps that I don’t sleep well. At this point I consider it a mortal risk.

That applies to vault as well.

0 comments

5 comments · 2 top-level

miyuru4y ago· 2 in thread

I also run 3 small clusters of consul and I went ahead and read the raft paper[1] so I can debug consul election problems if it occurs.

Consul is awesome when it works, but when it breaks it can be hell to get it working again. thankfully it usually works fine. I only had 1 outage and it fixed itself after restarting the service.

[1] https://raft.github.io/raft.pdf

rfoo4y ago

> so I can debug consul election problems if it occurs

Interestingly, reading this remind me of a HashiCorp Nomad marketing piece [1]:

> "We have people who are first-time system administrators deploying applications, building containers, maintaining Nomad. There is a guy on our team who worked in the IT help desk for eight years — just today he upgraded an entire cluster himself."

I was always thinking "but what if something goes wrong? just call HashiCorp engs?" :p

[1] https://www.hashicorp.com/case-studies/roblox

mrweasel4y ago

That seems to be a general problem with these types of solutions. You have the exact same issue with something like ZooKeeper. It awesome when it works, but good luck trying to figure out why it's broken.

Just the author of the previous post relying on these types of services is something that can keep me up at night.

darkwater4y ago· 1 in thread

> Quite frankly relying on consul scares the shit out of me. There are so few guarantees and so many pitfalls and traps that I don’t sleep well. At this point I consider it a mortal risk.

Consul (and Vault) for sure a complex softwares that 99% of the time "just work", but when they fail they can fail big time, I concur. But calling it a mortal risk seems a bit far fetched in my opinion.

hughrrOP4y ago

When you run 5 nines that’s a risk.

j / k navigate · click thread line to collapse