There's the machine. Then the VM. Then the container. Then the orchestrator. Then the controller. And it's all so complex that you need even more tools to generate the configuration files for the former tools.
I don't want to write a Kubernetes controller. I don't even know why it should exist.
The number of layers of abstraction you’re already relying on just to post this comment is nigh uncountable. Abstraction is literally the only way we’ve continued to make progress in any technological endeavor.
Then there are abstractions that may actually increase cognitive load "What if instead of thinking about chairs, we philosophically think about ALL standing furniture types, stools, tables, etc. They may have 4 legs, 3, 6? What about a car seats too?"
AFAICT writing a kubernetes controller is probably overkill challenge-yourself level exercise (e.g. a quine in BF) because odds are that any resource you've ever needed to manage somebody else has built an automated way to do it first.
Would love to hear other perspectives though if anybody has great examples of when you really couldn't succeed without writing your own kubernetes controller.
I'd argue the exact opposite has happened. We have made very little progress because everything is continually abstracted out to the least common denominator, leaving accessibility high but features low. Very few actual groundbreaking leaps have been accomplished with all of this abstraction; we've just made it easier to put dumb software on more devices.
What could have been a static binary running a system service has become a Frankenstein mess of opaque nested environments operated by action at a distance.
> I don't want to write a Kubernetes controller. I don't even know why it should exist.
You can take a look at Crossplane for a good example of the capabilities that controllers allow for. They're usually encapsulated in Kubernetes add-ons and plugins, so much as you might never have to write an operating system driver yourself, you might never have to write a Kubernetes controller yourself.
Mostly, because developers keep trying to replace sysadmins with higher levels of abstraction. Then when they realise that they require (some new word for) sysadmins still, they pile on more abstractions again and claim they don't need them.
The abstraction du-jour is not Kubernetes at the moment, it's FaaS. At some point managing those FaaS will require operators again and another abstraction on top of FaaS will exist, some kind of FaaS orchestrator, and the cycle will continue.
where do you keep the ECS service/task specs and how do you mutate them across your stacks?
How long does it take to stand up/decomm a new instance of your software stack?
How do you handle application lifecycle concerns like database backup/restore, migrations/upgrades?
How have you supported developer stories like "I want to test a commit against our infrastructure without interfering with other development"?
I recognize these can all be solved for ECS but I'm curious about the details and how it's going.
I have found Kubernetes most useful when maintaining lots of isolated tenants within limited (cheap) infrastructure, esp when velocity of software and deployments is high and has many stakeholders (customer needs their demo!)
ECS+Route53+ALB/ELB+EFS+Parameter Store+Secrets Manager+CloudWatch (Metrics, Logs, Events)+VPC+IAM/STS and you're pretty close in functionality.
I've never seen an infra/devops/platform team not swamped with work and just spinning their tires on random unnecessary projects. We're more expensive on average than devs, harder to hire, and two degrees separated from revenue. We're not a typically overstaffed role.
These days often DevOps is done by former Software Engineers rather than "old fashioned" Sys admins.
Just because you are ignorant on how to use AKS efficiently, doesn't mean your alternative is better.
> DevOps went from something you did when standing up or deploying an application, to an industry-wide jobs program. It’s the TSA of the software world.
DevOps was never a job title, or process, it was a way of working, that went beyond yeeting to prod, and ignoring it.
From that one line, you never did devops - you did dev, with some deployment tools (that someone else wrote?)
One of these gives you a way to democratize the knowledge and enable self-service across your workforce. The others result in tribal knowledge being split into silos all across an organization. If you're just running a couple of web servers and rarely have to make changes, maybe the manual way is OK for you. For organizations with many different systems that have complex interactions with each other, the time it takes to get a change through a system and the number of potential errors that manual tasks add are just infeasible.
Controllers are just one way to bring some level of sanity to all of the different tasks which might be required to maintain any given system. Maybe you don't need your own custom controllers, as there are a huge number which have already been created to solve the most common requirements. Knowing how to write them allows one to codify business rules, reduce human error, and get more certainty over the behavior of complex systems.
A bridge connects two otherwise separate geographical regions. To a government it's an abstract piece of infrastructure that will have economic and social impacts. To users it's a convenience that will change the way they plan journeys. To traffic planners it's another edge in a graph. To cartographers it's another line on a map. To road builders it's another surface to tarmac. To geologists it sits on a (hopefully) stable foundation that isn't expected to move or subside for at least a few hundred years. To cement people it's made of a particular blend that's the product of a specialised industry and expected to last for a hundred years. To metal workers it's reinforced with steel with particular strengths and weaknesses.
Nobody understands it all. Abstraction is not the source of complexity, abstraction is how we deal with complexity. The complexity is just there whether you want it or not. You think it's easy because you're the guy walking across the bridge.
Not all systems are small-N modern multi-tenant architectures deployed at small scale.
One example is a controller responsible for fulfilling ACME challenges to obtain x509 certificates. Something needs to actually publish the challenge responses somewhere on the internet, retrieve the x509 certificate, and then persist it onto the cluster so that it may be used by other applications. Something needs to handle certificate renewal on an ongoing basis. That something is the controller.
I don't want to write one either. Given the choice, I won't even touch one.
I think I know why they exist, though. Kubernetes is a system of actors (resources) and events (state transitions). If you want to derive new state from existing state, and to maintain that new state, then you need something that observes "lower" state transitions and takes action on the system to achieve its desired "higher" state.
Whether we invent terminology for these things or not, controllers exist in all such systems.
Abstractions are useful to stop 100000s lines of boiler plate code. Same reason we have terraform providers, Ansible modules, and well, the same concepts in programming ...
What do you do when site gets really popular and needs new copies? What happens when fill the VMs?If you want to automate it, that is a controller.
Also, if you are running on-premise, you don't need VM, you can use the whole machine for Kubernetes and containers for isolation. If you need more isolation, you can run VM containers; being able to switch is advantage of Kubernetes.
Why do this for relational databases? Why do I need to write a pg extension and SQL and an ORM when I can just write to disk?
It's certainly true that some use of Kubernetes is overkill. But if you actually need what it offers, it can be a game-changer. That's a big reason why it caught on so fast in big enterprises.
Don't fall into the trap of thinking that because you don't understand the need for something, that the need doesn't exist.
There is machine code. Then the assembler. Then the compiler, that targets the JVM. Then the programming language. Then classes and objects. And then modules. And then design patterns. And then architectural patterns.
Why all of this should exist?
...
Well, because each level is intended to provide something the previous levels cannot provide.
My last "operator" (not really an operator, but conceptually similar) is Airflow. Because Kubernetes doesn't have a way to chain job executions, as in "run this job after these two jobs finished".
If you're running your orchestrator on top of VMs, you're doing it wrong (or you're at a very small scale or just getting started).
In case of a CompositeController, the web service gets the created/updated/deleted parent resource and any already existing child resources (initially none). The web service then analyzes the parent and existing children, then responds with the list of child resources whose existence and state Metacontroller should ensure in the cluster. If something is left out from the response compared to a previous response, it is deleted.
Things we implemented using this pattern:
- Project: declarative description of a company project, child resources include a namespace, service account, IAM role, SMB/S3/FSX PVs and PVCs generated for project volumes (defined under spec.volumes in the Project CR), ingresses for a set of standard apps
- Job: high-level description of a DAG of containers, the web service works as a compiler which translates this high-level description into an Argo Workflow (this will be the child)
- Container: defines a dev container, expands into a pod running an sshd and a Contour HTTPProxy (TCP proxy) which forwards TLS-wrapped SSH traffic to the sshd service
- KeycloakClient: here the web service is not pure - it talks to the Keycloak Admin REST API and creates/updates a client in Keycloak whose parameters are given by the CRD spec
So far this works pretty well and makes writing controllers a breeze - at least compared to the standard kubebuilder approach.
The rendered manifest pattern is a simpler alternative. Holos [1] is an implementation of the pattern using well typed CUE to wrap Helm and Kustomize in one unified solution.
It too supports Projects, they’re completely defined by the end user and result in the underlying resource configurations being fully rendered and version controlled. This allows for nice diffs for example, something difficult to achieve with plain ArgoCD and Helm.
Is there runtime dynamism that you need the control loop to handle beyond what the built-in primitives can handle?
Other CRs are realized through imperative commands executed against a REST API. Prime example is KeycloakRealm and KeycloakClient which translate into API calls to Keycloak, or FSXFileSystem which needs Boto3 to talk to AWS (at least for now, until FSXFileSystem is also implemented in ACK).
For long-lived resources up-front (compile time?) expansion would be possible, we just don't know where to put the expansion code. Currently long-lived resource CRs are stored in Git, deployment is handled with Flux. When projects want an extra resource, we just commit it to Git under their project-resources folder. I guess we could somehow add an extra step here - running a script? - which would do the expansion and store the children in Git before merging desired state into the nonprod/prod branches, I'm just not clear on how to do this in a way that feels nice.
Currently the entire stack can be run on a developer's laptop, thanks to the magic of Tilt. In local dev it comes really handy that you can just change a CRs and the children are synced immediately.
Drawbacks we identified so far:
If we change the expansion logic, child resources of existing parents are (eventually) regenerated using the new logic. This can be a bad thing - for example jobs (which expand into Argo Workflows) should not change while they are running. Currently the only idea we have to mitigate this problem is storing the initial expansion into a ConfigMap and returning the original expansion from this "expansion cache" if it exists at later syncs.
Sometimes the Metacontroller plugin cannot be a pure function and executing the side effects introduces latency into the sync. This didn't cause any problems so far but maybe will as it goes against the Metacontroller design expressed in the docs.
Python is a memory hog, our biggest controllers can take ~200M.
The advantage of a controller is that it can react to external conditions, for example nodes/pods failing, etc. The is great for e.g. a database where you need to failover and update endpointslices. The advantage of a generator is that it can be tested easier, it can be dry-runned, and it is much simpler.
All of your examples seem to me like use cases that would be better implemented with a generator (e.g. Helm, or any custom script outputting YAML) than a controller. Any reason you wrote these as controllers anyway?
E.g. at one company I worked, they made a manifest to deploy apps that, in v1 was very close to Deployment. It felt owerkill. As they iterated, suddenly you got ACLs that changed NetworkPolicy in Calico (yes can be done with generator), then they added Istio manifests, then they added App authroizations for EntraID - Which again provisioned EntraID client and injected certificate into pods. All I did was add: this app, in this namespace, can talk to me and I got all this for "free". They code in the open so some of the documentation is here: https://docs.nais.io/explanations/nais/
One day, they decided to change from Istio to LinkerD. We users changed nothing. The point is, the controller was 2 things: 1: for us users to have a golden path and 2: for the plattform team themselves to have an abstraction over some features of kube. Although I do see that it might be easy to make poor abstractions as well, e.g. just because you don't create a Deployment (its done for you), you still have to own that Deployment and all other kube constructs.
I'm currently in a org that does not have this and I keep missing it every, every day.
Kinda like "functional core, imperative shell"?
We were using whitebox controller at the start, which is also like metacontroller that runs your scripts on kubernetes events. That was easy to write. However not having full control on the lifecycle of the controller code gets in the way time to time.
Considering you are also writing Python did you review kopf before deciding on metacontroller?
As we understood it, Kopf lets you build an entire operator in Python, with the watch/update/cache/expansion logic all implemented in Python. But the first operator we wrote in it just didn't feel right. We had to talk to the K8S API from Python to do all the expansions. It was too complex. We also had aesthetic issues with the Kopf API.
Metacontroller gave us a small, Go binary which takes care of all the complex parts (watch/update/cache). Having to write only the expansion part in Python felt like a great simplification - especially now that we have Pydantic.
On reflection though, I think this stuff can lead to a lot of complexity layers which don't benefit the product relative to the time investment. You are probably not Google.
I can’t find it again and i suspect the original author has deleted it.
One of the point for example was to go with ipv6 from the start, and another was about storage.
If anybody has a link, please paste it :)
But yes, the “public” kubernetes is a dumbed down version of borg. But frankly i think it’s mostly two things:
1. They probably had to redesign the things that in borg were too specific to the custom google infrastructure
2. They didn’t want to give away the _really_ good things, the competitive advantage.
Terraform running on a schedule gets you 3/4 of the way there for 5% of the complexity though.
Basically, declarative state implies value semantics which makes it easier to reason about. Underlying complexity is high though, and you need to judge how necessary it is.
I used a certain tool which had its own config format, and it's "cloudnative" operator implemented CRDs of which multiple can exist and they would update the config file in some mounted volume. Such thing is a hell to debug. Why can't we just store the config file in configmap/ secret and listen to changes?
(If we had a better templating solution than helm, I think quite a few operators wouldn't need to exist.)
isn't that what an operator also does basically? could you explain the problem with operators in more detail? thanks!
Either way I’m going to try my hardest to avoid this. K8s is hard enough to get right as is
