Couldn't agree with this more. At my last company there was a fair bit of time put into making the deployment process for our microservices as simple as added a new YAML file to the deployment repo. That file pulled a custom chart, and as a dev you just needed to configure a few variables to get everything up an running. But if you were deploying something that couldn't use one of the pre-configured charts it was a bit more work, especially if you've never done it be hand before.

Probably 98% of the devs were blissfully unaware of that complexity that the charts abstracted, and it let them focus on the services they were writing. I wasn't one of them, and always made sure to thank the devops team for simplifying the day to day deployments whenever I had to deal with writing a custom one.

THIS. In my homelab, I spent roughly a day cutting services over from Docker-Compose to Kubernetes. That day included writing the Helm templates for everything, bootstrapping three bare metal nodes with a hypervisor (Proxmox), clustering said hypervisor for HA on its own isolated network, making images and then installing K3OS onto six VMs across the three nodes (3+3 control plane/worker), installing and configuring persistent distributed storage (Longhorn) with backups to a ZFS pool as an NFS target (Debian VM configured via Packer + Ansible), configuring MetalLB, and smoke testing everything.

A day's work for one person to accomplish that isn't bad, IMO, but what that doesn't capture is the literal weeks I spent poring over documentation, trying things, running tests, learning what didn't work (Rook + Ceph is a nightmare), and so on. I went so far the day before the cutover as to recreate my homelab in Digital Ocean and run through the entire installation process.

Having services that magically work is hard. Having a golden path so you can create a new one with a few clicks is even harder.

This is still easier with Kubernetes than with other tools. Installing e.g. fluentd and Kibana is again a configuration / chart which you apply to the cluster. For monitoring and visualization again you have an operator / chart that you can apply to the cluster. Yes there is a lot of complexity and learning involved but overall I was still pretty amazed at how quickly I was able to get a cluster up to speed with the available tools.