For example using Vault v. Keywhiz [1] since I made that comparison earlier, this page feels like it's written to say "Vault does X. Keywhiz does too." That's not really what I'm looking for; I can diff the two projects' features in that way as well, but I want to know why it is that Vault is better, i.e., why is it that it exists, what sets it apart from other solutions I can use? Why should I invest my time and port my infrastructure into this project? Should I expect it to be around or abandoned in a year if there is no adoption?
I probably should have clarified this in my original post; sorry for the confusion.
From my limited perspective: Keywhiz is a system to take secrets, stored in a database, decrypt them with an HSM backed key, and expose them as files on a server. Because the secrets are exposed as a fuse filesystem, they require zero special code in your applications. Most other secrets stores just provide you with an API that requires custom integration work. You could probably even modify keywhiz-fs to talk to a Vault server.
On the other hand, Keywhiz doesn't know anything about the secrets inside it, so there's not much out of the box for secrets revocation, and other operations like that. There's also minimal secrets generation: We just provide an API to handle that. The biggest sources of secrets for us are service-to-service TLS certs, generated from our deploy system, which just calls the automation API to put the certs it gets from our internal CA. Keywhiz has a plugin system for generating secrets, but it's quite simple compared to what Vault has.
Vault also has some operational decisions that seem scary to me. They have a scheme for "unsealing" a vault with a set of secrets an operator has to input. This seems quite likely to cause an outage, while defending against an attacker you've noticed but who hasn't managed to steal the secrets out of your unsealed vault. While operating, your vault is unsealed, so an attacker who gets root has your secrets in both Keywhiz and Vault, but a service restart causes an outage. If you could somehow (via physical access, perhaps) boot a malicious OS onto a Keywhiz server, and your HSM doesn't defend against that, you could read secrets. For Vault, you also need to steal the keys to unseal. But those are accessible to humans, so that's probably easy.
As for lifetime, we don't expect Keywhiz to go anywhere. It's a crucial part of running Square. We've been using it for years now (though only open-sourced it earlier this year), and will probably continue to exist as long as the company does.
I'm happy to answer any other questions, too.
Unsealing doesn't cause an outage since you run them in HA-mode (side by side, leader elected).
As you said, if you gain root, both in Keywhiz and Vault, you can extract secrets at will. I can't comment to Keywhiz's capability (I'm sure some exists), but Vault will send audit logs to multiple backends so at least you can see it happening. The root token in theory can disable audit backends, but that action itself is also audited before it is disabled. That scenario in particular (root token compromise) is the worst case scenario. Ideally you throw that root token away into a N-key area as soon as possible. This feature is coming to Vault soon (expanding the shamir to arbitrary paths/secrets in Vault to support N-key operations).
Vault itself is used in places with very very high requirements of uptime (one user in particular can measure their overall downtime in seconds in the last quarter century). So we've had to go through some rigorous due diligence in such places to make sure it can withstand that. So, I assure you operationally that it won't cause an outage if it is properly configured in HA-mode. :)
I don't currently use either vault or keywhiz(we rolled our own solution using KMS and dynamodb) so I'm not really knocking either method. The current state-of-the-art with configuration management and secret distribution(particularly for container stuff) still requires a lot of customization code specific to each wrapped application, unless you wrote it all yourself to suite.
At the risk of sounding like a fanboy though, I will say that my own solution and others that have come out feel very itch-scratchy compared to vault. From what I have read about vault it has a much more thought out and productized vibe. Lots of cool features beyond delivering an encrypted blob from point A to B. I get this impression from a lot of Hashicorp components in general, and feel it's a shame larger open source projects like Kubernetes aren't integrating them while letting scaling to 200 hosts be a blocker to their next release... But I digress.
For config files with secrets, there's often a way to include another file, or sometimes we just drop the whole config file into Keywhiz.
Here's an example from the Keywhiz codebase, of a file that's loaded from disk, but in prod could be swapped for a properly secured secret https://github.com/square/keywhiz/blob/master/server/src/mai...
In development, you don't need to run Keywhiz, so you can just write a file on disk, so that's a nice advantage: Less things to depend on.
