undefined | Better HN

story

0 pointschucky_z5y ago0 comments

Nomad + Consul is the happy path. Adding Vault into the mix is nice, but not required.

Consul by itself is the game-changer. Even in k8s it's a game-changer. It solves so many questions in an elegant way.

"How do I find and reach the things running in (orchestrator) with (unknown ip/random port) from (legacy)?" being the most important. You run 5 servers, and a relatively lightweight client on everything (which isn't even outright required, but it sure is useful!), and you get a _lot_ with that.

Consul provides multiple interfaces and ingress points to find everything. It also is super easy to operate, and has a pretty big community.

If you absolutely cannot have Consul, Nomad is still a really good batch job engine, and makes a very great "distributed cron," which is more extensible, scalable, and easy to use than something like Jenkins for the same task.

My team is pretty small (was 4 people, now 6) and we manage one of the worlds largest nomad and consul clusters (there are some truly staggeringly large users of Vault so I won't make that claim). Even when shit really hits the fan, everything is designed in a way that stuff mostly works; and there's enough operator friendly entry points that we can always figure out the problem.

0 comments

jsmith126735y ago

Interesting, thanks for sharing!

I'm assuming your team is using vault for PKI, but is there a similarly happy path for issuing certs without Vault.

I started off just using `openssl` but it all felt very janky, and I didn't really have any idea how CRLs should be setup

chucky_zOP5y ago

Vault is great for just a PKI, even if you aren't using it for anything else. There are some tools that just do PKI, but Vault works a real treat at it. Any Terraform backend that supports encryption + Terraform + Vault gives you such an amazing workflow. We use a mix of short and long certs, with different roles based on what's getting a cert.

For now, we have CRLs disabled on all short-lived backends, enabled on long-lived backends and we're actually looking at disabling storing short-lived certs in the storage system at all, and just cranking the TTL down to really truly short. We've tested it as low as 30m, but a more real-world max-ttl is 1 week, with individual apps setting it as low as they can handle. For reference we run more than 10 PKI backends, and adding one (or a bunch) more is just a little terraform snippet for us.

The way it works via hashicorp template land, is that you just plop

    {{ with secret "name-of-pki/issue/name-of-role" "common_name=my.allowed.fqdn" "ttl=24h" }} {{ .Data.certificate }} {{ end }}

into your Nomad template stanza, or use consul-template directly as a binary, or use vault agent with it's template capability. You can get the CA chain if required the same way, just hitting a different PKI endpoint.

Also, as of Vault 1.4, Vault's internal raft backend is now production ready, making it a snap to run.

Try running through a few of the Vault quick-start guides, and replicating them in Terraform as much as possible. There's a few things TF does not handle gracefully last I checked (initial bootstrap), but you can get around that by using a null_resource or just handling that outside Terraform.

j / k navigate · click thread line to collapse