1) internal users: mainly developers by providing CI/CD
2) external users: end users
The platform of choice is mostly Kubernetes these days, which promises among other things stuff like
- load balancing
- self-healing
- rollbacks/rollouts
- config management
In particular, a main critic is "state" and the possibility to change that state by e.g. messing with config files via SSH, which makes running and maintaining these servers more error-prone.
However, my main question is:
"If this old way of doing things is so error-prone, and it's easier to use declarative solutions like Kubernetes, why does the solution seem to need sooo much work that the role of DevOps seems to dominate IT related job boards? Shouldn't Kubernetes reduce the workload and need less men power?"
Don't get me wrong, the old way does indeed look messy, I am just wondering why there is a need for so much dev ops nowadays ...
Thanks for your answers.
I agree with some other comments in this thread about a general fervor in the Enterprise space to "modernize" needlessly. This conversation usually lands on the company copying what everyone else is doing or what Gartner tells them to do. Cue "DevOps".
100 percent agree with your comments on something simpler. I can't tell you how many times I've debated with our Analytics teams to just use Docker Compose/Swarm.
I don't agree. I spun up a Kubernetes cluster in Azure, which was indeed easy. But then I had to figure out how to write the correct deployment scripts to deploy my docker containers to it, and how to configure all the security stuff. After more than a week of trying to figure it out, I decided to ditch the whole solution and go for Azure Container Instances instead. It was too much for me to learn about all the concepts of Kubernetes, how you configure them, how to make it work for solutions that are not as simple as the example on the website, and how to navigate through the various different methods of deploying stuff.
Maybe I'm just too dumb. But I wasn't going to invest a month of my time into doing something that should be simple enough for an average developer to accomplish.
Definitely a lot of work.
eksctl create cluster --name mycluster --region us-west1 --with-oidc --fargate --external-dns-access --asg-access --full-ecr-access --alb-ingress-access
The lower level infrastructure/platform/kubeadm type stuff isn't really 'Dev' related at all.
...until you need to debug something somewhere in the enormous stack.
This has been the biggest drain on my career. Everyone wants to be an "engineer" ready to handle every problem like its the next facebook. Like bruh, this service is going to get like 100 req/hour max and only when the sun is up - just **ing throw it on cloud run. We can tell the only thing you want to build is your resume.
But what would you suggest in lieu of Kubernetes? What would save work for a shop which is not yet a giant but has already overgrown the capabilities of 2-3 manually managed boxes / instances?
I can think of several options. Management by Ansible / Saltstack / Chef can easily become a rabbit hole comparable to maintaining K8s's YAML. In your experience, does Nomad save SRE work at smaller scales? Does Terraform? CloudFormation?
I run a small cluster for my side projects consisting of 3 nodes. 1 node is dedicated to run database containers and control plane node (it's also a worker node but strictly for database statefulsets only), and the other 2 as worker nodes, but one of them also run an NFS server that mounted as persistent volumes so every worker nodes can have access to it.
Self-healing? Internet-scale? I have on interest for them on my small cluster. I just want my apps to not go down while I'm updating them without writing complicated blue-green deployment scripts, ability to move around pods to other nodes when one got overloaded, and ability to add or remove nodes when needed without starting again from scratch. I basically treat it like docker-compose on steroid. So far it works really well.
Kubernetes only exists, because Google lost the Cloud Wars, and this was their Hail Mary pass.
If Kubernetes does indeed provide the best solution to provide scalability and availability, one can argue that this would result in a decreased demand for dev ops engineers, as they "would just have to use Kubernetes" and it would decrease their workload.
In reality this does not seem to be the case, that's why I asked.
I'd say it would result in either:
- the same scalability and availability with fewer DevOps engineers - better scalability and availability with a similar number or more DevOps engineers
In my experience, it's almost always the second case that happens. For example, a service would be moved from a few (virtualized or physical) servers that can only be scaled manually, to a k8s cluster with either autoscaling or at least scaling by changing a configuration file.
Pretty much every piece of software I've written in the past decade that tends to have configs in production can also work with little or no config. Including clustered software that just exploits things that are widely available like Zeroconf to get a decent low effort cluster up and running. No, you probably won't be able to (or want to) use those features in production, but that's beside the point. The point is to lower the thresholds. And then keep aiming at lowering them wherever you can because asking other people to care about your software is seriously uncool. Other people will never care as much as you do.
It is normal for programmers to become defensive about software. Be it their own or software they really like. But it is far more productive to assume that when users think something is awkward, perhaps it is because it is awkward. And perhaps it could have been done better.
Nobody actually gives a crap what someone thinks Kubernetes was built for -- and what kind of rubbish experience they think is deeply justified by the goals.
It is either needlessly awkward to use or it isn't. And guess what: most people think it is awkward. And I seriously doubt it needs to be this awkward.
k8s is not the right tool for every job. Most companies are not at the scale where they need to worry about the problems that it's trying to solve. But it's a cargo cult - they see the blog posts about how moving to k8s solves a bunch of problems that come up as you scale and decide they need to be solving all of those problems also even though there are simpler solutions at their current scale.
There's a bunch of other platforms out there that are way more opinionated and less "awkward" but they don't have the buzz that k8s has.
Of course, there's a huge cost to building your own framework as well... And it's easy to get wrong.
I started https://www.jetpack.io/ recently to try and build a better solution on top of Kubernetes. If you're interested in checking it out and giving us early feedback, you can signup on the website, or email us at `demo at jetpack.io`
Kubernetes is a great tool if you have scaling management problems.
"Things" here mean different things for different people. For me, it's secrets/certs, better rolling deploys and infra as code.
I have to manage like half a dozen docker images. K8s seems like a massive overkill, but managing by hand is rather error-prone.
yet its really not great at the first, and I'm strongly suspicious about the latter claim. Its very simple to make boneheaded decisions about networking that makes thing fragile.
In terms of scale, you have a limit of 5k nodes. given how fucking chatty it is, it costs a lot to run loads of nodes.
This is wrong, deploying on Kubernetes is easy and quick for most apps, you have one docker image one deployment spec and that's it.
https://kubernetes.io/docs/concepts/workloads/controllers/de...
If you need less than 8 instances to do host your product, run far away anytime anyone mentions k8s
I am consulting with a startup right now that chose to go everything docker/k8s. The CTO is half-shocked/half-depressed by the complexity of our architecture meetings, although he used to be a banking software architect in his previous assignments. Every question I ask ends up in a long 15 minutes monologue by the guy who architected all of it, even the most simple questions. They are soon launching a mobile app (only a mobile app and its corresponding API, not even a website) and they already have more than 60 containers running and talking to each other across three k8s clusters and half of them interact directly with third-parties outside.
Even as I am being paid by the hour, I really feel sad for both the CTO and the developers attending the meeting.
k8s is definitely not for everyone. Google has thousands hardware systems running the same hypervisor, same OS, same container engine and highly specialized stacks of micro-services that need to run by the thousands. And even, I am not sure that k8s would satisfy Google's actual needs tbh.
Ironically, there are some companies that highly benefit from this and they are not necessarily "large" companies. In my case, k8s and devops in general made my life infinitely easier for on-site trainings: those who come with a poorly configured or decade-old laptop can actually enjoy the labs at the same pace than every other attendee.
The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is like a movie set compared to the real thing.
for example: https://kubernetes.io/docs/setup/best-practices/cluster-larg...
no more than 5k nodes.
Its extra ordinarily chatty at that scale, which means it;ll cost on inter-vpc traffic. I also strongly suspect that the whole thing is fragile at that size.
Having run a 36k node cluster in 2014, I know that K8s is just not designed for high scale high turnover vaguely complicated job graphs.
I get the allure, but in practice K8s is designed for a specific usecase, and most people don't have that usecase.
for most people you will want either ECS(its good enough, so long as you work around its fucking stupid service scheme) or something similar.
K8s is a "middle 80%" scale solution. Its not made to run Google (though Google uses it a ton internally). Its also not made for your average four person startup (though, if you've got that experience internally, its not a bad choice; its not Heroku, but its better than a lot of deployment options out there).
All I'd say is: I've worked in a "scale-up" B2B org under $1M in ARR. We were pretty monolithic; just a backend NodeJS app, and a frontend SSR React app, basic. Five engineers by the time I left. We used K8s (EKS+Fargate). Maybe 50 pods total, across two environments. It was fantastic. We never had to say No to any weird customer, product, or engineering decision which would be difficult in either more managed, or more legacy, systems. Customer wants a custom domain and'll pay $50k for it? Like five lines of YAML and update Route 53, done. Datadog sidecar container so we can ingest some APM traces? Ten lines of YAML copy-pasted from their docs, done. Update the cluster? Click one button in the AWS UI. Every developer wants their own staging environment? Ok, bit more work, but: create some namespaces, retool the CI a bit, we can deploy separate databases in there as well its only staging, actually pretty straightforward.
Half the stuff we did with k8s would have taken three times as long with more native AWS solutions, and some of it probably would have been impossible on something like Heroku. K8s strikes a balance. Its not the simplest thing in the world. I wouldn't grab it on day 1 of a startup's engineering journey. But I wouldn't knock a startup who does.
The fundamental design just isn't infinitely scalable, and at a certain point, you might rather have some bulkheads/autonomy or regional diversity.
Anywhere I’ve worked business always prioritizes high availability and close to zero downtime. No one sees a random delivered feature. But if a node fails at night - everybody knows it. Clients first of all.
We’ve achieved it all almost out of the box with EKS. Setup with Fargate nodes was literally a one-liner of eksctl.
Multiple environments are separated with namespaces. Leader elections between replicas are also easy. Lens is a very simple to use k8s IDE.
If you know what you’re doing with Kubernetes (don’t use EC2 for nodes, they fail randomly), it’s a breeze.
There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter.
Holy shit, is this considered small to medium enterprise now?
With some exceptions, I believe that's a few orders of magnitude above what a small business can run on. Nowadays people just start their day by drinking some K2l-aid and spinning up a "basic" 6-node cluster for a development prototype.
Maybe I'm wrong, of course.
Forget a whole datacenter, even just one rack is an unimaginable amount of computing power, these days!
Is my observation correct that unicorns start to see that scale?
Folks, listen, if StackOverflow can run on this: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
So can your doctor's appointment website, your little ML app or Notion clone.
"But...". No. You ain't gonna need it. Do some load testing, prove it to yourself. Now, multiply the load by 100x, reserve AWS resources and you're good to go.
And with-k8s and without-k8s to me is pretty similar: we vendor or FOSS most of it. The major cloud vendors all have container registries (of … varying quality…); similarly, at a previous company we used S3+a small shim as a Python package store. (We later moved to a vendored solution.)
ELK for logs meant having a daemon set up per VM. Easier in k8s where I can push a DaemonSet to the entire cluster. With VMs … it's a per-app nightmare, really. Even then, that's really not perfect. In practice, in both situations, I feel like you end up having to integrate the apps with the metrics/logs providers. There's just not a common format. Sometimes, there are some libraries, e.g., there's some stuff for Prom's HTTP metrics APIs. Logs … eugh. Nothing amazing; getting structured logging requires per-app changes regardless of what you do. Sure, in either VM or k8s, you can just "suck up syslog/journald / docker logs", but what format are those in? They're not, is the answer, and I find most places do a "one text log per line" assumption (and then have stuff with multiline logs that just gets destroyed/corrupted/lost by the logging daemon) and it misses out on any sort of structured logs. jsonlines through those channels is a slight step up, but usually requires app changes.
It's also a Google engineer caliber solution. Lots of teams said “hey if Google engineers do it then it must be good!”…but forgot that they didn’t have the same in-house talent as Google.
The API is the main drawcard of k8s in the first place, if you are off in ECS land all you are doing is wasting a bunch of time on a dead-end.
I would instead focus on getting to understand the basics of the API by using a hosted k8s service like GKE or EKS. Stick with some basic manifests, i.e deployments, services, ingress.
Once you have some stuff running you can start learning how it really works and goes together, i.e what are pods, why are pods immutable, why is a replicaset, how does a deployment orchestrate multiple replicasets, what are endpoints, what is the difference between pod readiness/liveliness.
Don't cheat yourself this early in the game, just learn things the right way from the start and save yourself a bunch of work.
If you want to stay on the Kubernetes route check out k3s. Super easy to setup and usable for small production workloads
Before I used Kubernetes for my side projects and only at work I always thought it's hard to operate and very tricky. If you start with an empty "default" cluster and then just add bits when you need them it's actually not that complicated and doesn't feel too heavyweight. I'd suggest to just play around with a simple example and then see how it goes.
There's things that are used in "production" clusters that you don't need at the beginning, like rbac rules, Prometheus annotations etc.
There's now even onprem ECS variants which means not having to pay aws very much and still get the benefit of them running and maintaining the control plane
It's ridiculous overkill, but I'm looking at a NextCloud server on one of their PiBox hardware servers for the house. (You don't need a PiBox - their stuff will run fine on little instances from AWS/DigitalOcean/Hertzner, etc., or a spare PC you have lying around...)
> 1) internal users: mainly developers by providing CI/CD
> 2) external users: end users
>
> Nowadays we call people that do 1) DevOps and people that do
> 2) SREs (so one could argue that the role of sys admins just
> got more specialized).
SRE is a specialized software engineering role -- you'd hire SREs if you wanted to create something like Kubernetes in-house, or do extensive customization of an existing solution. If you hire an SRE to do sysadmin work, they'll be bored and you'll be drastically overpaying.
DevOps is the idea that there shouldn't be separate "dev" and "ops" organizations, but instead that operational load of running in-house software should be borne primarily by the developers of that software. DevOps can be considered in the same category as Scrum or Agile, a way of organizing the distribution and prioritization of tasks between members of an engineering org.
---
With this in mind, the question could be reframed as: if projects such as Kubernetes are changing the nature of sysadmin work, why has that caused more sysadmin jobs to exist?
I think a general answer is that it's reduced the cost associated with running distributed software, so there are more niches where hiring someone to babysit a few hundred VMs is profitable compared to a team of mainframe operators.
This makes a lot of sense. The same thing happened in the past with new technology, such as the electronic spreadsheet:
"since 1980, right around the time the electronic spreadsheet came out, 400,000 bookkeeping and accounting clerk jobs have gone away. But 600,000 accounting jobs have been added."
Episode 606: Spreadsheets!, May 17, 2017, Planet Money
Given that the US has transitioned out of manufacturing and into businesses services - I don't think much of this is explained by technology creating new jobs.
I think it's just explained by the workforce growing - and the shift in the US's role in the global economy.
Go back and read a few DevOps books and blogs by the founders of it. We will always need separate disciplines for dev and ops, just like we need mechanical engineers and mechanics/racecar drivers. But we need them to work together and communicate better to solve problems better and not throw dead cats over walls.
You can of course give devs more powerful tools, more access and agency to enable them to develop the software better. Sentry.io is a great example of what is needed; makes everyone's life easier, devs can diagnose issues and fix bugs quickly without anyone in their way. That doesn't require operations work because it's just simplifying and speeding up the triage and debug and fix and test phases. It's the fundamental point of DevOps.
You want the person who designs the car to know what a car is, and to be able to diagnose basic issues like "the fuel gauge says 'empty' and engine won't start". And there's no analogy to an Indy car driver in software, every distributed system is self-driving.
The most popular alternative to "DevOps" is a team of developers who do not run the software, and may not even have the skills or capabilities needed to boot up the server process. They do their development in an IDE, run unit tests to verify functionality, and do not have permission to log in to the production environment.
Meanwhile the "ops" side consists of people who may know basic shell scripting, or some Python if they're a go-getter, but are unable to read stack traces or diagnose performance excursions.
I agree with this definition of DevOps. However the vast, vast, vaaaast majority of real life uses of the term "DevOps" I've seen are just rebranded sysadmins. Sometimes it at least implies a more engineering approach to their coding. But in these institutions the Devs and Ops are very much separate groups of people, unfortunately.
If you're lucky the hiring manager might have at least read the Phoenix Project or the Google SRE books.
Holy smokes, did that thing blow up. A pod would go down, get stuck in some weird state (I don't recall what anymore), and K8s would spin a new one up. Okay, so it was running, but with ever-increasing zombie pods. Whatever. Then one pod would get in such a bad state that I had to nuke all pods. Fortunately, K8s was always able to re-create them once I deleted them. But I was literally deleting all my pods maybe six or seven times per day in order to keep the service up.
Ultimately, I rewrote the whole thing with a simplified architecture, and I vowed to keep clear of K8s for as long as possible. What a mess.
I think there was some jank on AWS CNI drivers at one point that delayed pod init, but that's probably the most wtf that I've personally bumped into thankfully.
Yes, and the unforgiving part of k8s is that there is a right way documented somewhere, you might have just spent 3 days sifting through docs and posts and community forums to find it.
It's sometimes worth it, sometimes not, my main gripe with k8s would just be that there is no "simple things" and it shouldn't be pitched as making it easier for small shops. Even if a small use case can be done elegantly, it will probably require a pretty comprehensive and up to date knowledge of the whole system to keep that elegance.
I think you're absolutely right. I freely admit that I knew NOTHING about K8s before embarking on this project (and still pretty much know nothing about it now), and I was able to cobble together something that 'worked', but that doesn't mean it was right.
And as another commenter points out, "a huge asterisk there if the tool makes it very easy to 'do it wrong'". I would rather be very clearly told that I've got it wrong and be prevented from progressing further vs. making something that superficially seems right then crashes and burns in prod.
I'm sure there are folks that can wield Kubernetes with great effectiveness, and good on them, but I found it to be supremely frustrating and the wrong tool for the right job. Not that I have a better solution, so I'm admittedly just kind of complaining.
All you need is to rewrite your application (think microservices), reduce cold latency (get rid of anything VM based such as Java, or rewrite in Spring or Quarkus), use asynchronous RPC, and decouple compute and storage.
Then you need an elastic platform, for instance Kubernetes, with all the glue around such as Istio, and Prometheus, and Fluentd, and Grafana, Jaeger, Harbor, Jenkins, maybe Vault and Spinnaker.
Then you can finally have your production finely elastic, which 90% of companies do not need. Microservices are less performant, costlier, and harder to develop than n-tiers applications and monoliths, and way harder to debug. They're just better at handling surges and fast scaling.
If what you want is:
- automated, predictable deployments
- stateless, declarative workloads
- something easy to scale
Then Docker Compose and Terraform is all you need.
If you also need orchestration and containers are your goal, then first try Docker Swarm. If you need to orchestrate various loads and containers are a mean and not a goal, then try Nomad.
Finally, if you will need most resources Kubernetes has to offer (kubectl api-resources), then yes, opt for it. Few companies actually have a need for the whole package, yet they have to support its full operational cost.
Most companies just pile up layers, then add yet a few more (Java VMs on top of containers on top of an orchestrator on top of x86 VMs on top of(...)), and barely notice the miserable efficiency of the whole stack. Well it's using Kubernetes, it's now "modernized".
Running a big Java monolith with 128GiB RAM footprint in Kubernetes works well. It's at its best when deployments are infrequent and traffic patterns are stable (or at least predictable).
If someone wants stateless microservices with immediate scale-up/scale-down, then that's more like a FaaS ("functions as a service") and they'll be better off with OpenFaaS instead of Kubernetes.
Every time I read a post like this about Kubernetes, I scratch my head. It takes me maybe half a day to deploy a CI/CD pipeline pushing into a new Kubernetes cluster with persistent DB's, configuration management, auto-renewing SSL certs and autoscaling API/web servers per environment. I'm by no means an expert, but I've been running 10+ sites this way for various clients over the past five years, with almost zero headache and downtime.
When I compare this solution to the mishmash of previous technologies I used prior to Kubernetes, it clearly comes out on top (and I use/d Terraform religiously). Setting up automatic server provisioning, rolling updates, rollbacks, auto-scaling, continuous deployment, SSL, load balancing, configuration management, etc... requires an incredible amount of work. Kubernetes either provides most of these out of the box, or makes them trivial to implement.
The only way I understand this argument is if you're building an extremely simple application. The nice thing about simple applications is that you can build them using any technology you want, because they're simple. Despite this, I often Kubernetes anyways, because it's _so simple_ to take a Helm chart and update the image name.
You don't even really need to do this, as you can tell k8s how to check if the pod is healthy, what a reasonable timeframe for becoming healthy is, etc. I've got some services which can take up to 10 seconds before they're actually ready to serve workloads and k8s can scale those services up and down without too much issue. Its definitely nice to reduce cold latency, but I wouldn't say you need to do it.
Can you elaborate on the deployment story here a bit?
A lot of this stuff really is trying to address the core problem we've had for a long time that probably won't ever end - "works fine on my computer."
But DevOps means many things because it's not clearly defined, which also makes it difficult to hire for. It's a "jack-of-all-trades" role that people somehow fell into and decided to do instead of more traditional software engineering.
Also, from what I've experienced from our internship program, CS programs are really bad at covering these fundamentals. Students aren't learning such basics as version control, ci/cd, cloud platforms, linux, etc.
Is that K8s, or that you've outsourced hosting, storage and stateful storage to cloud services?
I suspect that things are much easier to automate effectively, and the knowledge for automating things is much more common.
That's because those things are not part of Computer Science, or so I'm told. I got a degree in Software Engineering and regret nothing.
Good. Those aren't 'fundamentals' of 'CS'.
In the olden days of 10 years ago, most operations teams worked around the clock to service the application. Like every day there would be someone on my team doing something after hours usually multiple. Tools like Kubernettes, Cloud (AWS, GCP, Azure) have added significant complexity but moved operations to more of a 9 to 5 gig. Less and less do I see after hours deployments, weekend migrations, etc. Even alert fatigue goes way down because things are self healing. This is on top of being able to move faster and safer, scale instantly, and everything else.
Operations side used to be a lot of generalists admin types and DBA's. With today's environment, you need a lot more experts. AWS alone has 1 trillion services and 2.4 billion of those are just different ways to deploy containers. So you see a lot more back end roles because it's no longer automate spinning up a couple servers, install some software, deploy, monitor and update. It's a myriad of complex services working together in an ephemeral environment that no one person understands anymore.
The longer answer is: When you switch to Kubernetes, you are introducing _a lot_ of complexity, which, depending on your actual project, might not be inherent complexity. Yes, you get a shiny tool, but you also get a lot of more things to think about and to manage, to run that cluster, which in turn will require, that you get more devops on board.
Sure, there might be projects out there, where Kubernetes is the right solution, but before you switch to it, have a real long hard thinking about that and definitely explore simpler alternatives. It is not like Kubernetes is the only game in town. It is also not like Google invents any wheels with Kubernetes.
Not everyone is Google or Facebook or whatever. We need to stop adopting solutions just because they get hyped and used at big company. We need to look more at our real needs and avoid introducing unnecessary complexity.
Everything has become so abstracted away these days that performance isn’t even a consideration because people don’t understand the full stack. And I don’t mean “full stack” as it’s slung around these days. EG: doing 20 round trips to the database looking up individual records (as each is an object code-side), when you could just do one. Things like that are opaque & many devs wouldn’t even care or know, but a little bit of education can make a huge difference.
Because we're living in the stone age of DevOps. Feedback cycles take ages, languages are not typed and error prone, pipelines cannot be tested locally, and the field is evolving rapidly like FE javascript did for many years. Also I have a suspicion that the mindset of the average DevOps person has some resistance to actually using code, instead of yaml monstrosities.
There is light at the tunnel though:
- Pulumi (Terraform but with Code)
- dagger.io (modern CI/CD pipelines)
Or maybe the future is something like ReplIt, where you don't have to care about any of that stuff (AWS Lambdas suck btw).
Ironically, "DevOps" started as a philosophy that developers should be able to do operations, e.g. deploy, monitor their apps without relying on external people (previously called Sys Admins, etc). Yet, we're at a stage where the "DevOps" role has become the most prevalent one. IMO things have temporarily gotten slightly worse to get much better later.
From the productivity standpoint, it is not acceptable that a Machine Learning engineer or a Full Stack Developer are expected to know Kubernetes. Or that they need to interact with a Kubernetes person/team. It is an obstacle for them to produce value.
Kubernetes is not THE solution. It's just an intermediate step. IMO, in the long run there'll be very few people actually working with technologies like Kubernetes. They'll be building other, simpler tooling on top of it, to be used by developers.
You already named few examples. I can name few more:
- Railway.app
- Vercel
- Render
- fly.io
- probably many more under wayHard pass.
1) Kubernetes is an infra platform for the ops in DevOps. If developers need to spend a lot of time doing Kubernetes it takes away from their ability/time to do their dev. So, there are a lot of platform teams who pull together tools to simplify the experience for devs or DevOps specialists who handle operating the workloads.
2) Kuberentes is, as Kelsey Hightower puts it, a platform to build platforms. You need DevOps/SREs to do that.
3) Kubernetes is hard. The API is huge and it's complex. The docs are limited.
> 3) Kubernetes is hard. The API is huge and it's complex.
> The docs are limited.
The biggest challenge with learning Kubernetes is that third-party documentation is largely funded by ecosystem startups flogging their weird product. It can be very difficult to figure out things like network overlays when there's so few examples that don't involve third-party tools.
-------
Take the development of programming as an analogy:
* Punched cards
* Programming in assembler
* Goto
* Callable procedures
* Proper functions
* Compiled languages (There used to be companies just selling a big C compiler)
* Interpreters/JIT compilation/...
* ...
-------
And here's a similar progression:
* Servers in your basement
* Some server rented where you login via SSH
* Docker/Kubernetes/Clusters in the cloud
* Lambdas and other serverless solutions
* ...
As a sibling comment pointed out: We're still in the stone ages. Somewhere between punch cards and proper functions.
-------
To rephrase it in reversal: "Infrastructure as code has it backwards"
Right now, we manually partition our code, then provision infrastructure and then push our code to this infrastructure.
Instead, we should take a high level language that compiles to the cloud:
Just write your business logic and the the compiler figures out what clusters/services/event-buses/databases/etc to use; It will automatically partition the code, package, build, provision, push, update. And there's even room for something like JIT: Based on the load parts of your logic get, the compiler could switch databases. Also: Automated data migrations based on your code updates. But I guess we'll end up with a big distributed virtual machine that scales infinitely and completely hides the existence of servers.
There's already some glimpses of this future: No-code, the magic pulumi does with lambdas, several language projects that get rid of the file system and just store the AST in a DB, smart contracts where you pay for single computation steps...
-------
But back to the question: Kubernetes/AWS/etc is a lot of work because it's not really THE SOLUTION.
We already have the tools to have a cloud-scale application dynamically redefining and re-allocating its infrastructure. Give those tools some time to mature, and I'm sure the capabilities will be awesome.
A good example is the common database backend. A lot of groups have some RDS instance provisioned someway to somehow serve some frontend. Over time, the number of users grow and the RDS needs more resources provisioned for it. My last 4 jobs, it was always someone's job to check the provisioning or respond to an alert and bump the provisioning up to whatever the next rung was. In the not too distant future, this type of task will be handled by a mildly intelligent CDK application.
I don't see people wanting to give up control to get a platform like that any time soon.
That's valuable because, on the scale of large companies, it's much easier to hire "a network expert" or "a storage expert" or even "a Gatekeeper policy writing expert" than to hire a jack of all trades that can do all of these things reasonably well.
The corollary from this observation is that Kubernetes makes much less sense when you're operating at a start-up scale where you need jacks of all trades anyway. If you have a team of, say, 5 people doing everything from OS level to database to web application at once, you won't gain much from the abstractions that Kubernetes introduces, and the little that you gain will probably be outweighed by the cost of the complexities that lurk behind these abstractions.
Traditional deployment models also have ways to separate the network, from the security, from the storage, from the application stack. But it's typically done through strict change control processes.
I see many orgs choose to change to K8s because it offers improvements to operational tasks related to provisioning all of those changes, and speeds up the old change control gateways.
However, K8s is tuned to operate extremely profitably in organisations that need to make large numbers of changes to their infrastructure or software stack all of the time. And begins to break down at scale (at least from a cost point of view) compared to alternative solutions if organisations are not meeting minimum requirements for deployments a day.
IT orgs have chosen to adopt this huge piece of operations software that is itself fairly monolithic, and requires large amounts of upkeep and maintenance to keep it running smoothly and provide constant availability.
But even though they've adopted this new massive fixed cost in their IT operations. They continue to use old change control processes, often because restructuring old teams, that have traditionally made the company a lot of money, into new teams proves to be an incredibly risky exercise.
And so, the net outcome, is that their overall IT operations processes marginally improve at best. But they now have simply absorbed a new fixed cost on top of all their existing ones. What's worse, I see in some orgs, the additional cost pressure is being noticed, but they attribute (IMHO incorrectly) this cost pressure to lack of competence in the new technology (K8s), and begin a massive hiring spree to try and find specialists to better tune the technology stack. The solution, IMHO, instead should be to push hard on existing teams to simplify and downsize, and create incentives to interact with the infrastructure APIs more aggressively, letting Sysadmins/DevOps/SREs deal with faults, errors and failures the way they always have, but with the new fancy tools that let them work more efficiently.
The goal of Kubernetes is to improve the portability of people by introducing abstraction layers at the infrastructure layer - These abstractions can seem overly complex, but they are essential to meet the needs of all users (developers, operators, cloud providers, etc)
Before kubernetes in order for a developer to deploy an application they would need to (send email, create terraform/cloudformation, run some commands, create ticket for loadbalancer team, etc) - these steps would rarely be same between companies or even between different teams in the same company.
After kubernetes you write a Deployment spec, and knowing how to write a deployment spec is portable to the next job. Sure there are many tools that introduce opinionated workflows over the essentially verbose configuration of base Kubernetes objects, and yes your next job may not use them, but understanding the building blocks, still make it faster than if every new company / team did everything completely differently.
If you only have a single team/application with limited employee churn - then the benefits may not outweigh the increased complexity.
And the quality can differ a lot too. I used to think that k8s is not necessary, since our team has mastered both stateless and stateful app deployment on VM, with ansible calling aws/gcp API. Everything just works.
And then I joined another company, which has a hotchpotch of unorthodox terraform and ansible code and a homebrew service discovery layer, with frequent incidents in the early mornings, especially on weekends, when autoscaling of aws/gcp VM would fail due to a myriad of reasons.
With k8s, there is a minimum quality.
It's like asking "if the computer saves us all so much work, why do we have more people building computers than we ever had building typewriters"?
Something can "save labor" and still consume more labor in aggregate due to growth.
The number of single-server setups with kubernetes thrown in for added complexity and buzzwords I’ve found is way too dang high.
Kubernetes definitely achieves this goal well, and in a relatively portable way. But just like any other engineering decision, you should evaluate the trade offs of learning a completely new OS just to get a simple web site up, versus running a nginx instance with bunch of cgi scripts.
In a small organization, you can get away with a sysadmin running a Kubernetes cluster to enable that. In a larger org you'll need SREs as well as Operations Engineers to build and maintain the tools you need to enable the engineers.
Just like there are a plethora of programming stacks, there exists a ton of choices in implementing a software supply chain. Kubernetes is valuable to infrastructure engineers to use it to create these systems in a maintainable and reliable way.
Kubernetes helps with: networking setup, consistent deployments, task distribution. That's about it. It's more standardised than plain VMs, but you still have to deal with the other 90% of work.
Then, for designing, implementing and maintaining all these extra elements is why you need a devops guy. Also not mentioning how extremely fast things are moving in the cloud era.
The majority of SREs and sys admins I know don't really want to run Kubernetes. They'll do it, if that's what's called for, but it just adds complexity to trivial problems.
Developers want Kubernetes because it's quick to deploy containers, networks, load balancers, anything they want. That's fair, traditional hosting doesn't have much that provides that level of flexibility or that will allow you to get off the ground at that speed.
The issue is that as a platform Kubernetes isn't that easy to manage. It certain has improve, but it can be difficult to debug when it break. Nobody wants a 3AM page that a Kubernetes cluster is down. We've even seen companies say that they'll just stand up a new cluster if something breaks, because it's faster. Add to that the complexity of components deployed inside your cluster. As an SRE it absolutely suck to get the "hey the network's broken" from a developer, because that means that you now have to be an expert on WHATEVER load balancer or CNI they've decides to pull in.
As great as much of the stuff you can deploy in Kubernetes is, it's still harder to debug than an application running on a VM or bare metal server, with a "real" network.
Back in the day we cobbled together solutions out of different parts because it gave us a strategic advantage over monolithic commercial solutions. It was cheaper, but it was also easy to customize and fit to product & user needs. Yes configuration management was a nightmare, and it came back from the dead as Terraform, because instead of an OS with mutable state we now have a Cloud with mutable state. Docker and Packer and a few other solutions have fixed a lot of the mutable state issues, but databases are still flawed and SaaS is still just a mucky mess of unversioned mutable state and nonstandard uncomposeable poorly documented APIs.
With Kubernetes, we're back in the land of commercial monolithic products. Technically you can build it yourself and customize it all, but it's expensive and time consuming and difficult because of how many components there are tied together. It "gives you everything you need" the way the International Space Station does. Do you need a space station, or a barn?
People get so wrapped up in terminology. Declarative doesn't mean anything other than "not procedural"; it's not better than procedural, it's just different. Plenty of declarative things are a tire fire. Infrastructure as Code just means "there is some code that is committed to git that can manage my servers". A shell script calling AWS CLI is IaC. Doesn't make it a good solution.
You can't just install a piece of software and be done. That's the entire point of the DevOps movement, really. It's not about what you use, it's all about how you use it. Work with humans to figure out what will work for your specific situation. Use your brain. Don't just install some software because it's trendy and hope it will fix all your problems.
And Agile isn't Scrum, but once again a buzzword became the catalyst for a change that the buzzword isn't even "supposed to" represent.
It's our fault for never learning our lesson about buzzwords.
Yes, it's quite complicated.
No, an API to control a managed EKS/GCK cluster + terraform + Jenkins/Azure DevOps/etc. does not mean that magically the developer can 'just deploy' and infrastructure jobs are obsoleted. That's old AWS marketing nonsense predating Kubernetes.
There's a whole maintenance of the CI/CD factory and its ever demanding new requirements around performance, around Infosec requirements, around scale, and around whatever unique business requirements throw a wrench in the operation.
Sticking to ECS I guess is a valid point. What Kubernetes gives you is a more sophisticated highly available environment built for integration (Helm charts and operators and setups that when they work give you more levers to control resources allocations, separations of app environments, etc.)
And as an aside, I've been doing this for 20 years and long before Kubernetes, before Docker, hell, before VMs were used widely in production, I observed the developer mindset: Oh but it's so easy, just do X. Here, let me do it. Fast forward a year of complexity later, you start hiring staff to manage the mess, the insane tech debt the developers made unwittingly, and you realize managing infrastructure is an art and a full time job.
A story that is visible with many startups that suddenly need to make their first DevOps hire, who in turn inherit a vast amount of tech debt and security nightmares.
Get out of here with, it's just API calls. DevOps jobs aren't going away. It's just the DevOps folks doing those API calls now.
"since 1980, right around the time the electronic spreadsheet came out, 400,000 bookkeeping and accounting clerk jobs have gone away. But 600,000 accounting jobs have been added." Planet Money, May 17, 2017, Episode 606: Spreadsheets!
1. What people expect: less work needs to be done to get what you had before.
2. What people don't expect: more is expected because what used to be hard is now simple
So while it may have taken a few weeks to set up a pet server before and as a stretch goal you may have made your app resilient to failures with backoff retry loops etc. Now that's a trivial feature of the infrastructure, and you get monitoring with a quick helm deploy. The problems haven't disappeared, you're just operating on a different level of problems now. Now you have to worry about cascading failures, optimizing autoscaling to save money. You are optimizing your node groups to ensure your workloads have enough slack per machine to handle bursts of activity, but not so much slack that most of your capacity is wasted idling.
Meanwhile, your developers are building applications that are more complex because the capabilities are greater. They have worker queues that are designed to run on cheap spot instances. Your CI pipelines now do automatic rollouts, whereas before you used to hold back releases for 3 months because deploying was such a pain.
Fundamentally, what happens when your tools get better is you realize how badly things were being done before and your ambition increases.
> Kubernetes is a platform for building platforms. It's a better place to start; not the endgame.
https://twitter.com/kelseyhightower/status/93525292372179353...
Everywhere I've worked, having developers use and develop Kubernetes directly has been really challenging -- there's a lot of extra concepts, config files, and infrastructure you have to manage to do something basic, so Infra teams spend a lot of resources developing frameworks to reduce developer workloads.
The benefits of Kubernetes for scalability and fault tolerance are definitely worth the cost for growing companies, but it requires a lot of effort, and it's easy to get wrong.
Shameless plug: I recently cofounded https://www.jetpack.io/ to try and build a better platform on Kubernetes. If you're interested in trying it out, you can sign up on our website or email us at `demo [at] jetpack.io`.
- Kubernetes is very complex to setup
- It is not needed for many use cases
- It is (hopefully) not the defacto and standard for devops
- Load Balancing is already a solved problem way before Kubernetes. For many use cases, you don't need the complexity. Even things like Self Healing are kinda solved by AWS Auto Scaling for example.
- NOt every use case needs Kubernetes and its additional overhead/complexity
- Most importantly, devops is not "one size fits all" magic wand that Kubernetes or any other tool can solve. Various nuances to consider and hence you need DevOps as a role.
It helps standardize:
- deployments of containers
- health checks
- cron jobs
- load balancing
Is it same/similar across teams within and outside your organization.
If not, what would it cost to build consensus and shared understanding?
How would you build this consensus outside your organization?
For small organizations, one should do whatever makes them productive.
However, as soon as you need to standardize across teams and projects, you can either build your own standards and tooling or use something like K8S.
Once you have K8S, the extensibility feature kicks in to address issues such as:
- Encrypted comms between pods
- Rotating short lived certificates
However, if not K8S then, what alternative should we consider to build consensus and a shared understanding?
Our current platform is much more stable, has more features, and bigger than what the prev team did.
There are plenty of things you can't see like security or backup or scalability.
Backup were done app by app basis. Now you can do snapshots in k8s.
Security still is a mess. But now you can at least isolate stuff.
Scalability meant installing your application x times manually and configuring load balancer etc. Now you set it up per cluster.
Additional features you get with k8s: Auto scaling, high availability, health checks, self healing, standardization.
A lot of things got invented which lead to k8s like container or yaml.
Now with the operator pattern you can also replace admin and embed operational knowledge I to code.
Infrastructure was not ready to be controlled by code like this ever before.
If k8s is as amazing and time saving as you would imagine, you'd expect many companies to want to adopt it, so you'd expect there to be lots of job postings!
It's like saying "if computers are such time savers, why do so many companies hire people that have knowledge in computers". It's because this is a good tool that companies want to hire people with knowledge in that tool!
Now because of new possibilities, and new development they want to switch to Kubernetes to have that new possibilities everyone is talking about, and now you have to build many new containers, configure k8s, autoscalling etc... and developers don't know it (yet) and don't have time to learn it.
So lets hire a DevOps (me) that will do it ;-)
A lot of what is called DevOps goes into adapting software to this new mindset. A lot of that software is not written with best practices in mind, and likewise lots of tools are still in their infancy and have rough edges. I think it's fair to say some time and resources go into learning new ways of doing things, and it might not be the best choice for everybody at this stage to spend those resources unless there's an obvious need.
Kubernetes and 'DEVOPS' are the new hotness at non-FANG companies as they are always 5-10 years behind the trends. Expect to see more of it before it goes out of fashion.
Also DevOps is just a title. Nobody read the book, nobody is trying to create what the original guy at Google or whatever had in mind. It is just a all encompassing job doing the same activities that the sysadmin used to do. HR tells companies that they should rename their sysadmin departments to DevOps departments and everything else continues as normal.
1. Kubernetes isn't the solution 2. Kubernetes is expensive and extremely maintenance prone 3. Most of the companies I've seen switch to Kube I've seen switch away afterwards
Every time I've seen someone bring up Kubernetes as a solution, everyone at the table with first hand experience has immediately said no, loudly
Remember, there was a time at which someone wouldn't have been laughed out of the room for suggesting Meteor stack, and right now people are taking GraphQL seriously
Kube doesn't make sense until you have hundreds of servers, and devops makes sense at server #2
TL; DR - K8s and cloud let me do the work of ten of my old selves.
Many years ago I worked at a large, old tech company where all of these areas had dedicated teams.
PS: how many people at your company "really" know Kubernetes inside out? And if it misbehaves, who do you expect to have the answer?
When technology makes previous difficult things easy, this technology will be used to do more things that were previously impossible.
I personally haven't seen anyone use k8s to achieve things that were impossible before. They just use it because 1) They think everyone is using it 2) They don't know how to do it any other way
It's true that moving away from mutable state is a sea change that is (very) slowly engulfing the industry. But the tide is still very far out. The cloud is a pile of mutable state, leading to the same system instability and maintenance headaches that existed before the cloud, requiring new skills and workers to deal with. Redesigning the cloud to be immutable will take decades of re-engineering by vendors, and even when it's done, we'll still need economies of scale to reduce headcount.
Originally I thought it was a methodology assisted by a set of tools to make it easier for devs and ops (sys admins) to work together, mostly by giving both sides the same environment, usually in the form of docker containers.
Admins configure servers, or server pools since physical machines tend to be abstracted these days, set up applications, including the CI/CD stuff, make sure everything is secure, up to date and in working order, etc... Devs write the code, and test it on the platform supplied by the admins.
And now I see all these DevOps jobs. I mean what does it means? You are dev or you are ops, so what does DevOps means? It doesn't mean both, DevOps usually don't write software, so DevOps is ops, maybe we could call them container administrators, like we have database administrators.
I think that the confusion between the DevOps methodology and the DevOps job title give the wrong idea. Someone needs to make all these servers work, calling it serverless just means there is another abstraction layer, and while abstraction has some benefits, it rarely lessens the workload, but it may change job titles, here sysadmin -> DevOps.
Wages does not count as cost of goods sold, so who cares about a couple of extra hires? Funding is easy.
Also you severely underestimate the amount of work that goes under DevOps. Everything from build servers to test and security infrastructure is usually handled by DevOps. It's a massive surface area and it would be way worse without kubernetes.
Division of labor and specialization are two natural results of any rapidly evolving industry. The tech industry is no exception. The problem is that there are currently comparatively quite a lot of ways to learn the skills necessary to become a competent web developer, backend engineer, data scientist, etc. Compared to these titles, the ways to learn the skills involved in designing, operating, and maintaining scalable cloud infrastructure have not kept up with market demand.
Kubernetes is not "the" solution, but it is one of several solutions to the problem of standardizing a trade skill for the purposes of making it transferrable. Nobody wants to go to work at a place where the skills they need to do their job well are both difficult to acquire and completely useless once they get a new job.
Yeah, you lost me already. This is a bit like asking why there are other languages besides Java.
Now, I’m not saying k8s is as transformative as the word processor, but if it was, you’d probably expect to see more ops people, not fewer. They’d just be doing different things.
As part of my job, I see thousands of small and medium businesses happily embracing kubernetes. They do not have separate devops or security or infra folks. A few engineers do it all. Yes, there are challenges with so many moving parts and rapid releases with breaking APIs. But expect that to be fixed in coming years.
Why so many DevOps jobs? Are you noticing the jobs being eliminated, or are you just seeing more DevOps jobs being created?
A lot of companies moved to the cloud because their old data centers (or hosts) were driven by ticket systems instead of API's and access management. What took three weeks was now solved in seconds; the beginning was fascinating without a doubt. Then companies realized they owned none of this virtualized infrastructure and were at the behest of a very large corporation who could make sweeping changes with little to no notice. Although Kubernetes was pitched as providing extra grease to the gears of the enterprise, and they weren't wrong, that is not its total value to the enterprise.
The real value in Kubernetes running your own platform on someone elses hardware, especially to the degree where you can eventually free yourself from cloud provider lock-in that the above incurred. An example, if a company can spin up a team to create a database-as-a-service in Kubernetes clusters, then your RDS costs can shift dramatically down, and it develops a new level of capability and understanding that your company never had before.
I'm a SRE-SE, but I mostly use the title "Distributed Systems Software Engineer" because I feel that really fits what I do. DevOps is just a catch-all title for non-application-software tasks and roles at this point because it consumed so many things like "release manager", "QA", "application operations", etc... Personally, I do not trust companies or teams that use this as some sort of distinguishable title.
To answer the last part of your question, "Why are DevOps everywhere", because companies have diverse needs in terms of supporting software and software development, and DevOps is basically the catch all email of software engineering.
Read the introduction to the SRE book, available free online [1] - and you'll see that SRE is defined _in contrast to_ systems administration. Its specifically defined as software engineering with the goal of managing operational complexity.
Modern shops' failure to understand this (most SREs haven't read any of the book, let alone stopped to think what SRE actually means) is IMHO a primary factor in the failure of most "devops transformations"
For example, you don't need a docker container to deploy Mysql, although you can deploy Mysql inside one. But most development processors are so badly managed that one product has many conflicting libraries and dependencies. Eventual requiring each component to be isolated within its own container. Finally leading to an unmanageable number of containers requiring Kubernetes to manage the mess.
At the end of transitive reduction, you end up with a graph with all redundant dependencies removed but functionally, it is still the same graph.
Also, there is more demand than supply. Everyone wants to do Kubernetes and DevOps pipelines but the amount of folks experienced in those fields is small compared with demand.
It requires knowledge in many domains because it abstracts the entire data center. So you can’t just take a mid level sysadmin or developer and expect them to jump right in.
I have no data to back this up, but my hypothesis is that if you zoom out, and look across the entire industry, the % is vanishingly small. It may seem like every company is running or adopting kubernetes within our bubbles but our perspective is biased.
(Note: I'm not espousing an opinion on kubernetes itself, just about it's total adoption across the entire industry and how that effects the number of devops/sysadmin/SRE roles.
However, as with most large platforms, they require ceremonies and priests (devops engineers). Someone has to make the offerings.
Much as people would like to believe, you don’t reduce complexity, you just shuffle it around and there’s an exchange rate. Even with solutions like Fly.io, you’re not getting rid of complexity in aggregate, you’re paying them to manage it (I.e. the exchange rate).
I'd wager providing all those things 20 years ago without k8s and CI tools would've required relatively more sysadmins
Kubernetes buys the org some things but it is complex and you have to know how to write the app in a certain _way_ in order of the app to be "scalable, etc.".
There are no free meals or as someone smarter than me said long time ago "there is no royal road to learning".
The question is: do _you_ need this added complexity? That humble VPS can scale _a lot_ too.
1. Dethroning sysadmins introducing devops in the middle ("devs" capable of deploying in the cloud but unable to control the OS).
2. Increase CPU and other resource consumption (promoting heavy frameworks, unable to pass the Doherty threshold in 2022).
For clouds, increasing complexity and costs almost always expands the business.
What I'm trying to do at work is simplify the build ecosystem for all languages to the familiar `configure ; make ; make test ; make install` sequence that works well for OSS. If every ecosystem fit into that metaphor then the loop could be closed pretty effectively by any cloud provider by letting users add repositories to the CI/CD framework and it would do e.g. a standard docker build (configure, make, test), docker push (make install 1/2), and kubectl rollout to k8s at the end (remainder of make install).
Blockers:
Liveness and readiness checks are not automatic; they need to be part of each language*framework so that developers don't have to implement them by hand. At Google they just sort of came with the default HTTPServer class in your language of choice, with callbacks or promises you knew you had to invoke/complete when the instance was ready to serve. It helped that only 4 languages were officially supported.
Integration tests have no standard format and many deployments are combinations of artifacts built from multiple repositories, and configuration is not standardized (configmaps or ENVs? Both? External source of feature flags?) so all integration tests are manual work.
Metrics and SLOs are manual work; only humans can decide what actual properties of the system are meaningful to measure for the overall health of the system beyond simply readiness/liveness checks. Without key metrics automatic rollouts are fragile. This also means autoscaling isn't truly automatic; you need quality load metrics to scale properly. Not all services are CPU or RAM limited, and sometimes the limit varies depending on traffic.
All that said, cloud functions (Google, AWS, or other versions) are beyond DevOps. If you don't need high-QPS services then use cloud functions. They bypass 90% of the headaches of having code running on https endpoints. Most people don't have high-QPS (10K requests per second per shard) services, and could probably get away with cloud functions (1000 RPS on GCP). Everyone else pays the DevOps or hopefully the SRE tax for now. But we're still trying to automate ourselves out of a job; we don't want to be doing DevOps day-to-day either.
But like everything else in tech, solution of the problems at a given level enables everyone to do much more and build more complex systems that do more complicated stuff on top of that level. We always push the frontier forward with every new solution.
It’s not.
You still need Devops staff.
Cloud just provisions the hardware and OS. You still have to be responsible for the apps. You still have to be responsible for IO, memory, cpu and networking capacity.
You still need to make sure your apps are able to run on cloud - whether metal or k8s.
The solution to that is PaaS (Platform as a Service), and you can start a startup with almost no devops knowledge using things like Heroku and it’s myriad competitors from startups to AWS offerings.
Is it? These days I see SAM or Serverless Framework or other FaaS solutions all around me and it seems that everyone is migrating away from ECS/EKS/containers, it might be my own particular bubble though.
Luckily its from the functionally less evil google days and open source so it is possible to use anywhere.
Kubernetes is a powerful tool for "DevOps" roles. It provides an immense array of configuration, and largely replaces many OpenStack, Xen, or VMWare type environments. You can build powerful internal workflows on it to colocate many services on one compute fleet while maintaining developer velocity - which can translate to large margin improvements for some firms. This comes at a cost that you are likely to need a Kubernetes team, and potentially a dev tooling team to make it all work. On a large compute environment, the latter costs don't effect the big picture.
Now on the other hand, more teams than you would expect are just fine on Heroku/AppEngine/AppRunner/Lambda. These teams tend to pay the cost of not having a dedicated dev tooling team through more expensive compute, and sub-optimal tooling. The benefit here though is that "more expensive compute" may mean a fraction of a salary in many environments, and "sub-optimal" tooling may mean a production grade SaaS offering that has a few rough edges you talk to the vendor about.
IME it's much cheaper/lower risk to choose the latter in the long-run. The apparent savings from option 1 eventually turn into tech debt as the shiny tools get old, and migrating to newer/cheaper compute options becomes more expensive. I once built a colo facility which resulted in a 4x reduction in monthly recurring expenses (including salaries) for the same compute footprint, 1 year into the lifetime of the facility the former cloud provider reduced prices by ~30%. Around 6 months into the facility the DataScience team suffered attrition, resulting in fewer compute needs. At the 1.5 year mark the team begged for a flip to SSDs as they were having latency issues (a point of initial contention with the team that SSDs should have been used in the first place). Over the 3 year expected lifespan of the facility there were about ~2.5 months of ramp up/migration work which impacted ROI.
Overall, in hindsight, I'd say at best we achieved a 1.5x reduction in compute expenses compared to the alternative of tooling improvements, cloud cost reductions, and compute optimization. I now seek the tool which provides the lowest friction abstraction as at the worst case I can simply migrate to something cheaper - investing in compute infra has a crazy level of depreciation.
Eventually, people had a hard time finding well-rounded individuals that could design, develop, test, and deploy software. It seems to be a rare skillset, and people are resigned to not being able to hire for that kind of role. So, all of these ancillary concerns got split off into separate teams. You have a design team, a software engineering team, a test engineering team, operations, and so on. DevOps changed from "developers that operate their software" to "developer operations", which is just your 1990s operations team with a new name. You the developer want something, it goes on a backlog for some other team, you wait 6-8 years, you get your thing.
All the complexity of the devops world comes from having one team writing the software and one team running the software. An example are service meshes. They are super popular right now, and everyone and their mother is writing one and selling it for tens of thousands of dollars per year. To the software engineer, having two applications communicate over TLS is pretty simple; you read the certificates and keys from disk or an environment variable, throw them into a tls.Config, and give that tls.Config to your internal servers and internal clients. But, what happens in the real world is that the organization says something like "all apps must use mTLS by January 2023". The software team says "meh, we don't care, we'll get to it when we get to it". So the poor devops team is stuck figuring out some way to make it work. The end result is a Kubernetes admission controller that injects sidecars into every deployment, which provision TLS keys from a central server at application startup time. The sidecars then adjust iptables rules so that all outgoing connections from the original application go through the proxy, and if some distributed policy says that the connection is supposed to be mTLS, it makes that happen. Basically, because nobody on the dev team was willing to spend 15 minutes learning how to make this all work, it got bolted on by $100k worth of consultants, all for a worse result than just typing in a very small number of lines of code by yourself. That's the state of devops. The people writing the software won't run it, so you have to add complexity to get what the organization wants.
I think it's terrible, but that's the fundamental disconnect. When you need to change how software works without being able to edit the code, the workarounds get increasingly complicated.
As always, what looks like a software complexity problem is actually an organizational complexity problem. The way I've managed this in the past is to organize a lot of knowledge sharing, and make a conscious effort to avoid hiring too many specialists. At my current job my team used to make a SaaS product, and our team was a combination of backend software engineers, frontend software engineers, and some engineers with experience with operations. We were one team; frontend engineers would write Go code, backend engineers would make React changes, and we all did operational drills ("game days") once a week. The result was a very well-rounded team. Everyone could deploy to production. Everyone could be on call. Everyone could fix problems outside of their official area of expertise. I wouldn't have it any other way. The industry, however, deeply disagrees with that approach. So you're going to have testing teams, devops teams, etc.
Economics says that as something becomes cheaper, demand increases.
Essentially, these roles have narrowed the focus of system administrators and systems engineers. In one, you are concerned with CI/CD, and in the other you are making and maintaining cloudy solutions for people. This is yet another layer of abstraction for people, but it also means that most people do not know how to configure underlying software anymore. Because they lack knowledge of how to configure underlying software, they also require automation frameworks. They now do not know how to automate their workflows with Bash, Ruby, Python, or anything else. They need the cloud system to do it for them, which means that they get very vendor locked.
EDIT: the plus side of a new abstraction layer is cheaper tech departments at non-tech companies (fewer and cheaper personnel); which also means that pretty much everyone wants to be a software developer now, and very few people want to be SAs, SEs, DOEs, or SREs; you have to know everybit as much but you get paid much less.
All of this may bust. Increasingly, more and more people are becoming wary of monopolistic tech giants. The cost of their datacenters on the planet is increasingly rapidly. The governments of the world are growing wary of their increasing power. For businesses, complete reliance on a third party who has vastly more power isn't as palatable as it used to be. We may see a resurgence of smaller DCs and bare metal deployment, but any such change would only happen if another massive tech bust occurs. The reality that I see is that we may see both models live in tandem indefinitely, as there are differing use-cases that make either more suitable.
However, there are plenty of companies that will sell you a system, including varying levels of support. You then, of course, have to hire your own DevOps engineers that will deal with the areas the support doesn't cover, which, given the complexity, is still an awful lot. Or you do everything in-house, which means hiring even more people.
TL;DR DevOps engineers won't be out of the job anytime soon. Same for Kubernetes developers.