I have one. And it's managed. I don't think there's significant cost savings to going unmanaged, but maybe. Even so, why would I need a ton of them?
Can’t use cloud stuff on-prem and also if your clients have a server room of their own. Same for homelab.
Also it’s nice not to shift the pets attitude from servers to clusters and instead treat everything as cattle - provided you have backups of persistent data and the config versioned in a Git repo and there’s maybe some Ansible in the mix, being able to recreate an environment in the case of a fuckup is nice and also helps against bit rot.
Disclaimer: I actually prefer Docker Swarm/Compose over K8s due to simplicity (which matches my deployments and scale), but in the cases where I had to use a variety of K8s, going for K3s was pretty okay.
The cluster isn't that hard to recreate if things go south. Everything is in YAML configs already. And since I'm managed, it's just a few clicks in DigitalOcean to create a new cluster. And actually, I think I can create clusters through their CLI too, so if I did want to automate it, it's already ready to go. So I'd say I'm cattle-ready, but too cheap to pay for more cattle.
I nearly went Docker Compose/Swarm by accident when I was just getting started. I knew I wanted to dockerize my app but then couldn't figure out how to get it into prod. Then I found out people don't actually use Docker Compose for prod it seems and eventually stumbled into Kubernetes. It took a few weeks to wrap my head around, but I'm happy with it now.
Once you have a nice set up, I'd say it's pretty simple to maintain. DevSpace is fantastic for development, and then for deployment I just wrote a little script which builds my images, updates the kustomize with new images and applies the manifests. Pretty simple.
I run Minikube in Podman for dev. And then I use kustomize to customize dev, staging and prod environments. The environments are 99% the same, they just have different env vars and memory limits.
The only piece that's maybe a little dicey is the single load balancer/gateway. If there's a hiccup in that, then everything goes down.
But I've only blown up my cluster once in like 8 years or something, that's not too bad. It was a learning experience :-)
What other kinds of isolation do you want? I can see maybe a separate staging environment if you want to test gnarly things like that ahead of rolling them out to prod. And I guess maybe they can eat eachother's resources if you don't have request limits nor auto-scaling enabled.
But I'm cheap and managing more clusters sounds like a pain. Then I'd have to deal with more kubectl credentials and what not too.
How large is your operations team?
What you’re saying makes it sound like you’re a one-person operation, or somewhere close to that scale. That obviously doesn’t have same requirements as much larger organizations.
I ran a job last night which provisioned a cluster with 4TB of RAM and nearly 1000 vCPUs. It ran for 20 minutes, ingested about 800 GB of data from nearly an million files, and was then deleted. To do that on a single cluster that’s also used for serving production requests would be unnecessarily complex and risky. Our production system has users in every timezone using the system 24x7. At the very least you’d have to provision separate node pools anyway, but why would you bother to do that?
