I mostly agree with all of what you've said here. In our case, it's not unusual for a single customer environment to surge to 200-300 instances of an underlying compute server, and then scale back down to 20-30 at steady state. With 30 customer environments, you might have customers running from anywhere as low as 15 containers to as many as 500+, with a lot of dynamic flux depending on data ingestion and ETL.

K8S is in flux, so you still have to have a few top-end SRE types to manage your kube environment - the acceleration / maturity of the ecosystem is incredible though, so, sometime in the next 3-4 years, we'll start to see things get standardized enough that the wizardry required to keep it running will become a more commodity skill set.

And, more importantly, most of the ecosystem is fairly identical between azure/google/AWS - so porting or going multi-cloud is usually a weeks effort if that's something you want to do.

By "Moving up the Stack" - Of course I understand that cgroups/linux underpins it all - it's just that we're not using linux system binaries to manage the containers directly.

I mean tasks like process, storage, memory, CPU, resource utilization isn't something we tweak/query with OS commands, rather we're sending request/limit configurations to kube, and let it worry about managing the resources, relying on PromQL to monitor resource utilization, etc...