The response I always got when suggesting Kubernetes is "you can do all those things without Kubernetes"
Sure, of course. There are a million different ways to do everything Kubernetes does, and some of them might be simpler or fit your use case more perfectly. You can make different decisions for each choice Kubernetes makes, and maybe your decisions are more perfect for your workload.
However, the big win with Kubernetes is that all of those choices have been made and agreed upon, and now you have an entire ecosystem of tools, expertise, blog posts, AI knowledge, etc, that knows the choices Kubernetes made and can interface with that. This is VERY powerful.
I feel the same way about commercial APM tools. Obviously in a perfect world, you would have software so simple and fast that they’re unnecessary. Maybe every month or two someone has to grep some logs that are already in place. Once you’ve gotten yourself in situation where this is obviously not true, having Datadog, New Relic or similar set up (or using k8s instead of 100 unversioned shell scripts by someone who doesn’t work there anymore) will make your inevitable distributed microservice snafu get resolved in hours rather than a longer business-risking period.
Have they really? I have a few apps deployed on k8s and I feel like every time I need something, it turns out it doesn't do that and I'm into some exotic extension or plugin type ecosystem.
Something as simple as service autoscaling (this was a few years ago) was an adventure into DIY. Moving from google cloud to AWS was a complete writeoff almost - just build it again.
I'm sure it captures some layer of abstraction that's useful but my personal experience is it seems very thin and elusive.
Yep. Kubernetes is not just kubernetes when moving between clouds, it becomes a very opinionated product (for better or worse) with lots of vendor addons. Could someone that is familiar with one pick up on the other? Sure! But there are gotchas. And then kubernetes on prem adds the hardware lifecycle piece, and potential data locality issues, etc.
Yep! I am now using k8s even for small / 'single purpose' clusters just so I can keep renovate/argo/flux in the loop. Yes, I _could_ wire renovate up to some variables in a salt state or chef cookbook and merge that to `main` and then have the chef agent / salt minion pick up the new version(s) and roll them out gradually... but I don't need to, now!
I've used it every day since then so I have the luxury of knowing it well. So the frustrations that the new or casual user may have are not the same for me.
It really feels like we are drowning in self-imposed tech debt and keep adding layers to try and hold it for just a while longer. Now that being said, there is no reason not to add Kubernetes once a sufficient overlap is achieved.
And there's zero setup. Just a deployment yaml that specifies exactly what you want deployed, which has the benefit of easy version control.
I don't get why people are so bent on hating Kubernetes. The mental cost to deploy a 6-line deployment yaml is less than futzing around with FTP and nginx.
Kube is the new LAMP stack. It's easier too. And portable.
If you're talking managed kube vs one you're taking the responsibility of self-managing, sure. But that's no different than self-managing your stack in the old world. Suddenly you have to become Sysadmin/SRE.
It's not obligated to be, but it's also obvious why people would want it to be.
I think I disagree with this, or at least the implication. I think it is true you can solve EACH OF THISE PROBLEMS INDIVIDUALLY in a simpler way than Kubernetes, the fact that you are going to have to solve at least 5-10 of those problems individually makes the sum total more complicated than Kubernetes, not to mention bespoke. The Kubernetes solutions are all designed to work together, and when they fail to work together, you are more likely to find answers when you search for it because everyone is using the same thing.
I think it is fair to say k8s is not a zero cost abstraction, but nothing you use instead is going to be, either, and when you do run into a situation where that abstraction breaks, it will be easier to find a solution for kubernetes than it will for the random 5 solutions you pieced together yourself.
Docker and k8s are just wrappers around namespaces, cgroups, file system ACLs, some essential cli commands, which can also be configured per user.
We may be headed back there. Have seen some experiments leveraging Linux kernels BPF and sched_ext to fire off just the right sized compute schedule in response to sequences of specific BPF events.
Future "containers" may just be kernel processes and threads... again. Especially if enough human agency looks away from software as AI makes employment for enough people untenable. Why would those who remain want to manage kernels and k8s complexity?
Imo its less we agreed on k8s specifically and more we agreed to let people use all the free money to develop whatever was believed to make the job easier; but if the jobs go away then it's just more work for the few left
Docker, yes, but kubernetes is way more than that the instant you have more than one physical machine node. (If you only have one node in any deploy, sure, it's likely overkill, but that seems like a weird enough case to not be worth too much ink.)
If you silently replaced all my container images with VM images and nodes running containers with nodes running VMs, I think the vast majority of all my Kubernetes setup would be essentially unchanged. Heck, replace it all with people with hands on keyboard in a datacenter running around frantically bringing up new physical servers, slapping hard drives in them, and re-configuring the network, and I don't think the user POV of how to describe it would change that much.
"Kubernetes is beautiful. Every Concept Has a Story, you just don't know it yet... So you use a Deployment... So you use a Service... So you use Ingress... So..."
(Full text at the link)
I do run a k3s cluster for home stuff... But I really wish I could get what it provides in a much simpler solution.
My dream solution would effectively do the same as k3s + storage, but with a much simpler config, zero yaml, zero choices for components, very limited configuration options, it should just do the right ting by default. Storage (both volume and s3), networking, scale to zero, functions, jobs, ingress, etc... should all just be built in.
This comes down to that fact that software architecture is not great in general and very few people care about simplicity.
And then they hit all the things that make sense in big company with like 40 services but very little in their context and complain that complex thing designed for complex interactions isn't simple
The database is more complex since there is storage affinity (I use cockroachDB with local persistent volumes for it) - but stateful is always complicated.
No argument there. The Toyota 5S-FE non-interference engine is a near indestructible 4 cylinder engine that's well documented, popular and you can purchase parts for pennies. It has powered 10 models of Camrys and Lexus and battle proven. You can expect any mechanic who has been a professional mechanic for the last 3 years know exactly what to do when it starts acting up. 1 out of 4 cars on the road have this engine or a close clone of it.
It's not what any reasonable person would use for a weedwhacker, lawnmower, pool pump or an air compressor.
Now the actual CI/CD/thing-doers tools that all suck... I'm still stuck with those.
Same with build.sh and doing it in such a way that I can use all the build.sh in my ci.yml for Github Actions.
Some Kustomize, a little bit of envsubst and we're good to go thank you very much.
I manage 100+ variations on a single helm chart and 50+ such helm charts at work daily for 7 years across 11 datacenters/kubernetes clusters. And I have team members who swear by kustomize. The number of kustomize typo errors and issues that I have deal with is unimaginable. Whereas if I test and deploy a helm chart, I know it will work everywhere in every variation.
Kustomize is just plain terrible and backwards as a solution. It doesn't scale, it is half assed. It tries to basically require you to build your own compiler and parser and transform. With kustomize + envsubst: dear friend, you have built helm.
Fun times.
For very simple deployments, you don't need anything at all. Just write manifests and use `kubectl apply`. You can write `deploy.sh` but it'll be trivial.
If you want templating, there are many options. You can use `sed` for the most simple templating needs. You can use `cpp`, `m4`, `helm` or `kustomize`. I, personally, like `kustomize`, but `helm` probably not the worst template engine out there.
Kustomize is even somewhat included into basic kubernetes tooling, so if you want something "opinionated", it is there for you. It works.
Git and Kubetnetes configuration cannot go hand in hand. You cannot go back in past indefinitely because cluster state might not be that reversible. If so, git is useless.
And no, doesn't apply for database migrations. You can mostly run migrations backwards if each migration was written carefully.
I have a "simple" representation of services using CUE, that generates the yaml manifests and flux deploys them.
I hesitated a while before going the k8s route but before that I had a overly error-prone Ansible configuration and I got sick of manual templating (hence the move to CUE for type safety).
There's also the fact that I wanted my services to be as plug and play as possible, so for example automatically generated openid credentials and very easily configurable central SSO, along with the easily configurable reverse-proxy.
If anyone thinks that k8s is not the best tool for this, I'm always interested in advice.
(Also a lot of complexity in my setup is due to self hosting, I have Istio, MetalLB, proxmox CSI, and all other kind of stuff that your cloud provider would already have, and these are the things that take most of the configuration files in my repo)
(I tend to think this one is acceptable in the beginning, but certainly doesn’t scale.)
K8s is well suited to dynamically scaling a SaaS product delivered over the web. When you get outside this scenario - for example, on-prem or single node "clusters" that are running K8s just for API compatibility, it seems like either overkill or a bad choice. Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
There are also folks who understand the innards of K8s very well that have legitimate criticisms of it - for example, this one from the MetalLB developer: https://blog.dave.tf/post/new-kubernetes/
Before you deploy something, actually understand what the pros/cons are, and what problem it was made to solve, and if your problem isn't at least mostly a match, keep looking.
It’s well suited to other things as well, people are just in denial about some of them.
“I need to run more than two containers and have a googleable way to manage their behavior” is a very common need.
What's the problem with a single-node cluster? We use that for e.g. dev environments, as well as some small onprem deployments.
> Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
Which batteries are not included? The "wrapper around the underlying cloud provider services and APIs" is enormously important. Why would you prefer to use a less well-designed, more vendor-specific set of APIs?
I seriously don't get these criticisms of k8s. K8s abstracts away, and standardizes, an enormous amount of system complexity. The people who object to it just don't have the requirements where it starts making sense, that's all.
What surprises and gotchas did you have to deal with using k3s as a Kubernetes implementation?
Did you use an LB? Which one? I'm assuming all your onprem nodes were just linux servers with very basic equipment (the fanciest networking equipment you used were 10GbE PCIe cards, nothing more special than that?)
You can have one template docker-compose.yaml file and separate deployment files for different envs, like: docker-compose.dev.yaml, docker-compose.prod.yaml
I think swarm is really underrated
Yes (if I'm getting your question right). here is an example of nfs volume: https://docs.docker.com/reference/compose-file/volumes/#driv...
The people advocating for boring tech generally aren't interested in containers.
You can just run programs.
If your app is just a blob that can be run it is fine, but many languages make it more complicate.
I wonder if just putting app into .appimage + using systemd for some of the separation would be a sweet spot ?
If you had said "unikernels" I would have had no arguments to make.
I have no interest in going back to the hell that is deploying a java app.
- People who would prefer their way of doing this, whether that's deployments on VMs, or use some sort of simpler cloud provider.
I had the same opinion a few years ago, but have kind of come to like it, because I can cleanly deploy multiple applications on a cluster in a declarative fashion. I still don't buy the "everything on K8s", and my personal setup is to have a set of VMs bought from a infrastructure provider, setup a primary/replica database on two of them, and use the rest as Kubernetes nodes.
- People who run Kubernetes at larger scales and have had issues with them.
This usually needs some custom scaling work; the best way to work around this if you're managing your own infra[1] is to split the cluster into many small independent clusters, akin to "cellular deployments"[2]/"bulkhead pattern"[3]. Alternatively, if you are at the point where you have a 500+ node cluster, it may not be a bad idea to start using a hyperscaler's service as they have typically done some of the scaling work for you, typically in form of replacing etcd and the RPC layer through something more stable.
- People who need a deep level of orchestration
Examples of such use cases may be to run a CI system or a container service like fly.io; for such use cases, I agree that K8s is often overkill, as you need to keep the two datastores in sync and generate huge loads on the kube-apiserver and the cluster datastore in the process, and it might be often better to just bring up Firecracker MicroVMs or similar yourself.
Although, I should say that teams writing their first orchestration process almost always run to Kubernetes without realizing this pitfall, though I have learned to keep my mouth shut as I started a small religious war recently at my current workplace by raising this exact point.
[1] Notice how I don't say "on-prem", because the hyperscaler marketing teams would rather have you believe in two extremes of either using their service or running around in a datacenter with racks, whereas you can often get bog-standard VMs from Hetzner or Vultr or DigitalOcean and build around that.
[2] https://docs.aws.amazon.com/wellarchitected/latest/reducing-...
[3] https://learn.microsoft.com/en-us/azure/architecture/pattern...
For example:
* For some cursed reasons you want to make sure every single one instance of a large batch job see just one NIC in its container and they are all the same IP and you NAT to the outside world. Ingress? What ingress? This is a batch job!
* Like the previous point, except that your "batch job" somehow has multiple containers in one instance now, and they should be able to reach each other by domain.
I see docker as a way to avoid having a standard dev platform for everyone in the company so that the infra team don't have to worry about patch xyz for library abc, only run docker.
But, with all the effort put in place to coordinate docker, k8s and all the shebang, isn't it finally easier to force a platform and let it slowly evolve over time?
Is docker another technical tool that tries to solve a non-technical problem?
K8s is a way to take that and make it scale up for a large number of applications on a large number of hosts in a production business in a way that's automated and resilient to failure.
I run K8s at home. I used to do docker-compose - and I'd still recommend that to most people - but even for my 1 little NUC with 4vcpu / 16Gi Homelab, I still love deploying with K8s. It's genuinely simpler for me.
If anyone's looking for inspiration, my setup:
* ArgoCD pointed to my GitLab repos
* GitLab repos contain Helm charts
* Most of the Helm charts contain open-source charts as subcharts, with versions set like (e.g.) `version: ~0` - meaning I automatically receive updates for all major version until `1`
* Updating my apps usually consists of logging into the UI, reviewing the infrastructure and image tag updates, and manually clicking sync. I do this once every few months
My next little side project: Autoscaling into the cloud (via a secure WireGuard tunnel) when I want to expand past my current hardware limitations
The needed tweaks, the ability to customize things, basically goes to zero because the support staff is technical about the software, but NOT about Kubernetes.
I am not joking: a recent deployment required 3x VMs for Kubernetes, each VM having 256 gigabytes of RAM; then a separate 3x VMs for a different piece. 1.5TB of RAM to manage less than 1200 network devices (routers etc. that run BGP).
No one knew, for instance, how to lower the MongoDB (because of course you need it!) resource usage, despite the fact that the clustered VMware install is using a very fast SSD storage solution and thus MongoDB is unlikely accelerate anything; so over 128GB RAM is being burned on caching the results coming back from SSDs that are running at many-GB/s throughput.
So unless I want to dig into their YAML files that are not documented…
Kubernetes was overkill (I do that all day, 5 days a week); Kamal was too restrictive, so I found myself rolling out Yoink. Just what I need from k8s, but simple enough I can point it to a baremetal machine on Hertzner that can easily run all my workloads.
- using Tailscale SSH is brilliant
- using caddy-docker-proxy for ingress is brilliant
What do you use for:
- service discovery
- secret store (EDIT: Crap you use Infisical. No shade, I just have this horrible foreboding it will end up like Hashicorp. I use Conjur Secretless Broker but am tracking: https://news.ycombinator.com/item?id=47903690)
- backing up and restoring state like in a DB
PS: Have you been having issues with Hetzner the last few weeks?
For secrets, I self-host Infisical on the box -- easy to plug in whatever secret manager, should make it pair nicely with https://github.com/tellerops/teller or something similar
Had no problems with Hertzner so far, just enjoying the raw CPU power of bare metal. The plan is to roll out more boxes across different providers, using Tailscale for the backplane network and Cloudflare to load-balance between them. All in due time What issues have you been having ?
And that's a very important distinction when it comes to maintaining complex systems. This could've changed with LLMs (I'm still adjusting to what new capabilities mean for various decision-making logic), but before machine intelligence debugging an issue with Kubernetes could've been a whole world of pain.
Scaling is a sidenote, that it becomes easy is a result of hoisting everything else onto one control plane and a set of coherent APIs.
One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db.
I stepped in and told him he was about to build a data warehouse. :P
If your work is easily googleable/parseable by an AI, why would anyone pay you?
As to the job security idea: the only people who do this are people who aren’t good at creating real value, so they have to try to create niches where they’re needed.
I'm not doing overlay networks, I'm using a single bare-metal host, and I value the hands-on Linux administration experience versus the K8s cluster admin experience. All of these are reasons I specifically chose not to use Kubernetes.
The second I want HA, or want to shift from local VLANs to multi-cloud overlays, or I don't need the local Linux sysadmin experience anymore? Yeah, it's K8s at the top of the list. Until then, my solution works for exactly what I need.
But really this applies to any powerful tool. If you need to measure a voltage, an 4 channel oscilloscope also probably seems too complicated.
Dear friend, you have built a Kubernetes - https://news.ycombinator.com/item?id=42226005 - Nov 2024 (277 comments)
Basecamp is running successfully with Kamal and surely making more money than all the Kubernetes experts combined on this platform.
Small companies should not behave and pretend like big enterprises. Maybe Kubernetes is justified in those scenarios.
But for 95% of the population, it's an overkill and a constantly moving target.
Ended up doing a mix. Built on compose for now but in a manner that’ll lift and shift to k8s easily enough. Its containers talking over network either way
> Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of CommonLisp.
>> The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.
-- Donald Knuth, Computer Programming as an Art (1974)
EDIT:
> Except if you quit or go on vacation, who will maintain this custom pile of shell scripts?
Honestly? I don't care. There is a reason why I quit and 99% of time it's the pay. And if the company doesn't pay me enough to bother then why should I? Why should I bother about some company future in the first place?
And somehow k8s tends to use one of the most valuable chips. The smartest guy who from then on only works with maintaining and updating k8s. Instead of building the best and smartest products.
Kafka also competes with this.
