For example, if you use GKE (Google Cloud's K8s offering), you attach your K8s cluster to an auto-scaling node pool and it handles (de)provisioning of your VMs for you. You essentially don't care about the VMs, there's essentially no overhead.
If you are in a private cloud, this also creates a good "API boundary" between the team responsible for running hardware, and the team responsible for shipping software to run on that hardware. On the former side you can essentially just adopt a machine into the cluster and leave it, and on the latter side K8s lets you programmatically reference resources, but you don't need to know how/where they came from.
Yes, you don't need Kubernetes to come up with your own implicit or explicit API boundaries, and these might not be needed for smaller projects. I agree that Kubernetes is often used where it's not strictly needed.
There are things which strict abstraction, and with it, separation of concerns provide. The crucial point is that certain things are enforced.
You're right though, the abstraction layer is very interesting for _enabling_ public clouds, and for private cloud team ownership.
You also know that you can throw away VMs because they don't contain any state. You are not losing data just because you kill a VM or a VM breaks.
It is way easier to just spin up n nodes and provision them all equally than whatever you did before.
In my team, we can manage way way way more nodes than we ever could. We spin up 100 VMs and destroy them on a regular basis automatically.
Gardener for example supports autoscaling on bare metal. The whole ecosystem is providing tons of great options.
Lauding Kubernetes because ops work took too much of your time is just shifting your burden elsewhere, even if that means paying a bit more for an offering like ECS fargate.
Any environment with configuration management can treat instances as ephemeral. It’s a best practice.
I view docker more as a package manager, no more dependency hell.
In any event K8s is sprawling, it will soon be too complex for its own good. Assuming it’s not already.
I dont have anything against VMs. Feel free to click yourself a VM on any cloud provider, use it however you like.
K8s abstracts VMs away and i have and had real issues with maintaining VMs. Docker filling up the node with logs. Unable to upgrade the BaseOS due to python dependencies. Managing the same VM stack through ansible and everything ansible or chef brings to the table.
There have been plenty of self healing mechanism in place which do solve unfortunate issues. Memory? The service restarted, was offline for 3 minutes and is now working again. Node disk full? Pods get scheduled away, new node comes up, done. Update/upgrade of nodes? Nodepool does it for me.
For me, k8s has 2 real issues like memory (swap support is wip finally!) and stateful workloads like a database. But the concept of an operator shows a bright future.
k8s also does one thing very nice: It enforces certain aspects which are a pain in the ass later. That VM which wasn't updated for years and run just fine? Now there is an issue and it needs to be fixed asap. But now the debian repositories are no longer available. I have to fix apt srce list first, then i need to fix dependencies and then i need to restart it.
