It is year 2021, 1000s of new startups around kubernetes, more features, more resource types. Pods would still get stuck in terminating state or container creating state. It still pretty painful.
EKS continues to be painful. It has gotten better over the years, but it is a chore compared to GKE. I like to imagine that Jeff Bezos walked into someone's office, said "you're all fired if I don't have Kubersomethings in two weeks", and that's what they launched.
https://kubernetes.io/docs/tasks/run-application/force-delet...
I have seen few scenarios where people patching a statefulset actually screws up the volume mount. It is sometimes not evident where the error is for instance if it is the CSI or in the scheduler unless you deepdive into the issue.
so im guessing here that spotify wrote their own becouse they had a spesific requirement?
nomad was also early days in 2015
Can you elaborate? Does Kubernetes add some thrills to the relatively simple Docker network configuration?
https://kubernetes.io/docs/concepts/cluster-administration/n...
And yes... Kubernetes network configuration is on a whole different level from docker networking.
Things like VXLAN-EVPN are supported on paper, but are no where near mature compared to offerings from normal networking vendors. Heck, even the BGP support inside kubernetes is lacking. Which is a great shame because it creates a barrier between pods and the physical world. (Getting a VXLAN VTEP mapped to a kubernetes node is a major PITA for instance).
Most major cloud providers seem to have fixed this by building even more overlay networks (with the included inefficiencies).
Cost wise it was cheaper for a while but now RedHat are bumping up licensing costs so now I think is about the same costs.
Overall it seems like a waste of time, but has been interesting.
- Builds with fixed dependencies that never change. Rollback is easy
- Easy deployment of a prod environment on a local machine
- Fast deployment
- Easy automation (use version X with config Y)
With Kubernetes (or other derivates like Openshift) you get:
- Auto scaling
- Fail over
- Better resource usage if multiple environments are executed
- Abstraction of infrastructure
- Zero downtime deployment (biggest point for my company, we deploy >3 times per week)
There are applications that do not need Kubernetes or even containers, but is this list really nothing oO?
I can imagine that if you use Kubernetes just like a classic cluster it could seem like an unnecesarry added complexity but you gain a lot of things.
When someone writes "fixed dependencies" I read "developers can more easily add more bloat before the cardhouse tumbles". That happens for example when the "fixed dependencies" are upgraded.
I am miserable having to touch all this junk. I feel a project is right when I can just git clone it (a few megabytes of data at most) and am left with a self contained repo that was written with minimal dependencies (optimally stored in-tree), and that can be easily built in seconds with a simple shell script on any reasonably modern system.
The bare bones way takes a good amount of initial work, but mostly it's a learning experience. Once one understands a few principles of writing portable software, I'm sure it saves a huge amount of time compared to adding all these shells of junk.
--
Oh yeah, I have zero experience about integrating with Kubernetes or whatever. I've been a small time user of Jenkins and CircleCI (unvoluntarily), and when I don't have to set it up and it actually works, it's alright and can help where the developer maybe lacks a bit of discipline (build all targets, run all the tests).
But, I doubt these technologies are a replacement for an ergonomic build environment (with simple python build script or even a crude Makefile). Is incremental building a thing on any of this CI pipelines? Because one thing I want is building really really fast, and it's already way too much overhead if I have to go through a git commit to check this stuff. Don't even think about requiring a full rebuild or Docker image build just to get some quick feedback on a code change.
- Builds with fixed dependencies that never change. Rollback is easy -> what about VMs?
- Easy deployment of a prod environment on a local machine -> yep, that's a nice touch, the only valid point for me!
- Fast deployment -> lol no, Im faster with VMs.
- Easy automation (use version X with config Y) -> valid for VMs and baremetal too
With Kubernetes (or other derivates like Openshift) you get:
- Auto scaling -> you can get it with VMs too
- Fail over -> you can get it with VMs too
- Better resource usage if multiple environments are executed -> you can get it with VMs too
- Abstraction of infrastructure -> Should I really write it?
- Zero downtime deployment (biggest point for my company, we deploy >3 times per week) -> We do on some specific DC (government style) and we release 10-15 times and day with Bare metal servers and ansible
There are applications that do not need Kubernetes or even containers, but is this list really nothing oO? -> None of the arguments convinced me
I can imagine that if you use Kubernetes just like a classic cluster it could seem like an unnecesarry added complexity but you gain a lot of things. -> yes, extra cost and extra skills needed
> Builds with fixed dependencies that never change. Rollback is easy
Any good build system already did this, such as Bazel, or a Gemfile.lock. We'd just snapshot AMIs to keep OS dependencies fixed... which is what Docker images effectively do. If you re-docker-build the same Dockerfile, it's not like you get the same result of "apt-get install libxml" the next time either.
> Easy deployment of a prod environment on a local machine
How containers are deployed varies wildly between prod and the local machine. All the things that were hard before are still hard. Things like secrets and external dependencies still usually vary.
If prod is a kubernetes environment, getting a suitable k8s environment setup locally sucks, especially since it will probably have a different ingress controller, load balancer setup, storage classes available, resource requests, etc. If prod is kubernetes and local is docker-compose, that honestly seems like just as much work to create a second way to run the stack than just using a bash script + "npm start" or "bundle exec rails server" or whatever.
Either way, it's not really a prod environment. It's hard to run identical-to-prod environments locally, and those problems are related to secrets and clouds and such, not due to the lack of containers, in my experience.
> Fast deployment
In my experience, containers haven't sped up deployment. Let's say you use ubuntu for your host and container's OS. Before containers, this meant you had to download one version of libssl ever, and that was it. If there was an update to libz, that didn't require a new download of libssl. After containers, if you build your container for app1 last week, and your container for app2 today, the "FROM ubuntu" likely resolves to a different image. Both your apps now have different "ubuntu" layers, which probably have the same version of libssl, but deduplication of downloads only happens if the whole layer is identical.
In essence, we went from downloading 1 copy of libssl (for the host OS only) to 3 copies (host OS + 2 containers w/ different ubuntu bases), and there's no deduplication.
That by itself seems like it has to be slower since there's an inherent increase in network bandwidth that has to happen. Even if you have a shared base image, you're at least doubling the downloads of libssl since before you could use the host's copy only.
All the items you listed under k8s are things I had before it, excluding "Abstraction of infrastructure". Frankly, if you have a well-made load balancer, it's hard not to have zero-downtime deployments and auto-scaling.
I've seen more success from organizations running smaller K3s or K8s clusters (if they need the orchestration) or just running small apps via Docker/Docker Compose separately, using a CI system (even as simple as GitHub Actions) to manage deployments.
I wasn't around when it was introduced so can't talk to the initial complexity/pain but the current team responsible for it is surprisingly small.
The responsibility for everyday use has been delegated to all teams which gives them a capability to build and deploy frequently. That's a real enabler.
Most of the things that make this work are not really technology based, good collaboration, sharing knowledge and practices and being able to ask and get help quickly but I do think Openshift itself isn't bad at all. It does appear to provide a very decent build, deploy and run platform.
Was there particular problems, or just "all bad"?
kubectl create configmap foo --from-file foo.json -o yaml --dry-run | kubectl replace -f -Docker really used to crash a lot back in the days, mostly due to buggy storage drivers. If you were on Debian or CentOS it's very likely that you experienced crashes (though a lot of developers didn't care or didn't understand the reasons the system went unresponsive).
There was notably a new version of Debian (with a newer kernel) published the year after my experience. It's a lot more stable now.
My experience is that by 2018-2019, Docker had mostly vanished as a buzzword, people were only talking about Kubernetes and looking for kubernetes experience.
edit: at that time Docker didn't have a way to clear images/containers, it was added after the article and follow up articles, I will never know if it was a coincidence but I like to think there is a link. I think writing the article was worth it if only for this reason.
I don't know if we are doing it wrong at my job but... I feel like Kubernetes is just a way to orchestrate Docker images running as pods (amongst many other things I'm sure).
I know Kubernetes doesn't require Docker images but... how many shops are using k8s without Dockerfile build + tag + publish of images for containers?
I know that there are posts from 2016 explaining how to do so, but the article had no mention of this?
It made little difference, the entire ecosystem was highly unstable. Containers could still fail, the docker daemon could hang, and the host could kernel panic any minute. For databases that meant downtime and potential data corruption.
Besides, there is a major use case to run temporary databases for CI testing. I remember a lot of issues when running performance tests or seeding the database with initial data, basically anything that is performance intensive. I think the unstable filesystem played a role but it was far from the only root cause.
Honestly it's been 6 years now and I've never seen a company running any critical databases inside docker (but I've seen it for testing). I've known a lot of companies that said they did or they had plans to move existing databases, but that wasn't real. At the end of the day there was no sysadmin/DBA/devops who would do it, they understood that it would come back to bite them (many had enough troubles just running ephemeral web services). Maybe it's the harder part to grasp but database is really a different mindset from web development, you cannot risk losing customer data, this would be an extinction level event for your job and for the company.
Folks need to start building systems that destroy all and re-image fresh. Any other way you are just setting your self up for failure.
This whole "cattle not pets" charade gets on my nerves. Yeah, it's easy to "scale" that which is stateless. Not so fast with stateful. I don't care that I can spin up gazillions of web server instances. My data store is still one and very much stateful.
For many database needs, there are great options that natively support a highly available cluster.
For those needs that don't have a good clustering option, there are great network storage systems that are easy to deploy and use.
You don't need to treat hardware as a pet to have a database with persistent identity.
Simple solution is storage is a secondary drive/partition -- still automate OS install.
In a few years, the things businesses want to pay $$$ will change. New blog articles about "this new stuff is bad!" will appear, and new job postings paying above-market $$$ will appear also. You can either rail on about the bad(or good) changes, and how it's just everything-old-is-new-again....or you can get with the program, and get paid. In another few years, rinse and repeat.
It's interesting that this misconception made it into a clearly knowledgeable article. Containers have state on the writeable layer that is persisted between container stops and starts.
I usually start mine with —-rm so they’re removed on shutdown.
I’ve seen people apply security updates via ‘apt update; apt upgrade’ within a running container. Guess what happens when that container is eventually destroyed?
Honestly I think there is a lot to be said for making the writable layer of a container read-only. It makes sure that things like logging, if you care about them, go somewhere safe, or if you don't, get turned off explicitly. And also prevents gotchas like "oops wrote important data to /var/lib/notavolume when I meant to write to /var/lib/therightvolume" that show up at the worst times.
Sure there's issues (albeit a lot fewer as time progressed) with docker but for what it gained in productivity and developer's sanity, it was very welcome.
It's no surprise that implementing pretty much any process would be a massive improvement from that.
The more interesting question is whether containers offer any advantages for people who already have a (probably significantly more lightweight) build and deployment process.
The complaints about their development practices are valid (and haven't really improved), but even then the technology worked well so long as you understood its limitations.
"If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area."
- Yes, "clean old images" was not implemented back then. His hack is not that bad, and one can filter out in-use images if they want to pretty easily. Anyway, docker does have "docker image prune" now.
- Storage driver history discussion is entirely incorrect. No, docker did not invent overlayfs nor overlayfs2. There was a whole big drama of aufs not mainlining, but it was mostly in context of live cd's, not docker.
But the big missing thing is: you should not store important data in docker images, Docker is designed to work with transient container. If you have a database, or a high-performance data store, you use volumes, and those _bypass_ docker storage drivers completely.
- The database story is completely crazy... judging by their comments, they decided to store the database data in the docker container for some reason and got all the expected problems (unable to recover, hard to migrate, etc....). It is not clear why they didn't put database data on the volume, there is a 2016 StackOverflow question discussing it [0].
Also, "Docker is locking away [...] files through its abstraction [...] It prevents from doing any sort of recovery if something goes wrong." Really? I did recovery with docker, the files are under /var/lib/docker in the directory named with guid, a simple "find" command can locate them.
- By default, Docker uses Linux networking and yes, the configuration is complex so it adds overhead. That's why there is --net=host option (which was there for a long time) which just bypasses that all.
[0] https://news.ycombinator.com/item?id=12872636
[1] https://stackoverflow.com/questions/40167245/how-to-persist-...
Docker gradually exhausts disk space on BTRFS Open ghost opened this issue on 23 Oct 2016 https://github.com/moby/moby/issues/27653
Still comments this week showing it happens still.
However one of the things that i haven't quite understood, is why people use Docker volumes that much in the first place, or even think that they need to use additional volume plugins in most deployments?
If it's a relatively simple deployment, that has some persistent data and it's clear on which nodes the containers could be scheduled (either by label or by hostname), what would prevent someone from just using bind mounts ( https://docs.docker.com/storage/bind-mounts/ )?
And if you need to store it on a separate machine, why not just use NFS on the host OS to mount the directory which you will bind mount? Or, alternatively, why not just use GlusterFS or Ceph for that sort of stuff, instead of making Docker attempt to manage it?
For example, Docker Swarm fails to launch containers if the bind mount path doesn't exist, but that bit can also be addressed by creating the necessary directory structure with something like Ansible - and then you're not only able to not worry about volumes and the risk of them ever becoming corrupt, but you also have the ability to inspect the contents of the container storage on the actual host. Say, if there are some configuration files that need altering (seeing as not all of the containerized software out there follows 12 Factor principles with environment configuration either), or you just want to do some backups for the data that you've stored in a granular fashion.
A history of re-submitted, previously discussed posts:
Turned out, AWS had decided to upgrade Docker on the server and that version (20.x) is not able to launch services in the Swarm. We have downgraded to 18 now, which works, but is not a long-term solution.
But honestly after we used k8s we everything just works. Although I realise that GKE is actually moving to containerd for newer clusters? Not sure what made the decision, but last time I had to restart a container manually due to my stupid mistake, the api doesn’t seem to be much different