undefined | Better HN

0 pointsjpgvm7y ago0 comments

Translating something from docker-compose into k8s isn't generally hard. But yeah CFN is a nightmare, I wrote a bunch of tools to try make it suck less and contributed to stuff like cfndsl and I could still never get it to the point that I could have developers use it without an abstraction layer.

k8s definitely helps with the "all my apps stuff in one place that is easy to see". There are some pitfalls to avoid though. Don't use helm, it's bad for your health. Avoid deploying your own k8s cluster unless you really need to, just use GKE. Avoid custom resource definitions unless they are well supported, migrating off them can be hard - prefer tools that look at annotations etc (like external-dns, cert-manager and friends).

Of course things get difficult when you need statefulsets to run things like Kafka/ZK and friends but it's definitely possible and they run well once setup.

In my mind k8s is the only option right now that doesn't result in man-years being wasted on pointless AWS bullshit.

0 comments

sklarsa7y ago

Why not use helm? I'm looking into spinning up (and eventually productionizing) a k8s cluster at my job and I was leaning towards using helm since some pieces that I was thinking about using are installed via helm charts (https://github.com/kubernetes/ingress-nginx for example)

solatic7y ago

If you use third-party Helm charts, you eventually need to add onto the generated objects in a way which the chart doesn't support, and then you're up a creek without a paddle. This is precisely the use case which Kustomize tries to fix and it was the only real strength which Helm had to begin with (i.e. the ease of installing third-party software on your cluster).

In the meantime, because you must install Helm into every namespace in your cluster into which you desire to install charts, it's a massive resource hog and security risk. Charts themselves also need to be hosted somewhere, so you end up needing to install Chartmuseum, Harbor, or Artifactory (if you didn't have Artifactory already), and they have their own operational costs.

navd7y ago

I thought it would be useful to add that you can also generate k8s manifests from a helm chart using the `helm template` command.

I'm in the same boat where I avoid helm if at all possible.

1 more reply

alexandre_m7y ago

Something not pretty, but that works well for me, is using yarn to manage dependencies on upstream Helm chart repositories, then use kustomize to override certain things if required.

That checkouts a local copy locked at a specific version, which you can bump easily, and allows overriding the template definitions on your side.

No need for additional infrastructure.

j / k navigate · click thread line to collapse