They were not wrong on saying that Kubernetes was very complicated, at least in some sense. In the beginning no one wanted to use it because they could easily setup Docker Swarm with minimal effort. This argument still pops up frequently on HN when there's a new post about Kubernetes.
I guess the problem was they didn't realize why Kubernetes needs to be that complicated. And if a system is complicated for good reasons, that's actually good business opportunity and tons of people and companies will be willing to make the effort to fill that gap.
You can read Docker Swarm's docs end-to-end in < 30 minutes, and have a cluster running in roughly the amount of time it takes to install Docker on three boxes.
Docker's failing was a failure to market Docker Swarm. Most people don't even know it exists, yet every single machine with Docker has it already! The last time I checked, they only had a single dev that was part-time allocated to supporting it.
So it's not so much that Docker failed to jump on the k8s bandwagon, so much as they built a wonderful thing that they never marketed and semi-abandoned, to focus on Docker Hub and other core Docker things they thought they'd be able to more directly monetize.
Fortunately, Docker Swarm remains a wonderful piece of software, even without particularly active investment, because it is a fairly complete offering for most use cases. And it seems to be growing in adoption.
Swarm might be great, but the ecosystem around k8s is massive. So my two choices are: ignore Docker entirely and do it like I've always done (Heroku, chef, puppet, etc. just managing my servers myself), or buy into containers and go with K8S.
To that extent, I only ever deal with K8S at work, and for my personal stuff I don't go near Docker at all. Why would I when I can get an 8GB/4CPU VPS for $9/mo and serve everything I care about from that? I've got fail2ban, systemd, caddy, and plenty of resources.
I can host mysql on the same machine because there isn't a god-given law about hosting your DB on a separate box. Same with anything else.
No need for Docker or Swarm or K8s.
- deploying k8s on bare metal is not easy. It was never really designed for this and to be fair Google has a vested interest in making you use GCP. EKS is nothing more than cobbled together scripts and you still need to do a lot of manual work
- k8s introduces a ton of terminology and new concepts (pods/service/deployment/ingress) not to mention the whole host of addons built on top like Istio.
It shouldn't take any decent dev more than a couple days to install minikube/k3s or use a cloud provider, play around and get comfortable with k8s.
- pick up Swarm or Nomad now, and then potentially be blamed for the expensive and costly migration to k8s - pick up k8s now, waste the time of your (voiceless) devs and ops on an overengineered solution, but retain access to the larger pool of technologies and a smooth upgrade path should the need arise
Docker Swarm remains far simpler to set up, run, and troubleshoot than Kubernetes.
I just can't choose it over K8s, though, knowing that it's unsupported.
Borg is great if you have Google problems. K8s is not Borg.
That said, if your tooling or products or software supports kubernetes, which a lot more are these days, then it becomes a lot easier. A lot of companies these days are saying "Here's how to install it, or just grab this helm chart"; Gitlab, for example. We even had one piece of software we were testing which only had two options: cloud-hosted, or self-hosted on k8s. Nothing else was supported.
k8s doesn't seem to be any easier to configure, with the possible exception of microk8s (which has its own issues), but the amount of things you can do with it once it's up and running is compelling.
It is very simple to spin a minimal distro based cluster. It is relatively simple to have it up and running your solution. Sure getting Kubernetes from the source and tailoring all the required components to run cluster with full HA etc' is a complex stuff, but building a private cloud based on vanilla Kubernetes isn't the only or the most common use case.
Kubernetes has become a speeding bullet, it won't be easy to catch and it has to run through a lot of walls to slow down for a competitor to do any damage to it's dominance.
Ha, I can at least assure you we're not oblivious. Nomad has a product manager from Mesosphere and most of HashiCorp's products integrate deeply with Kubernetes. We're well aware Kubernetes is the juggernaut to which every other scheduler has succumbed.
I believe there's room for both Nomad and Kubernetes. Whether as competitors or complements, having more than one vibrant project in the orchestration space will hopefully make all the projects better for users. Any one project has to make tradeoffs that improve the user experience for some while degrading it for others. For example Kubernetes has an IP-per-Pod network abstraction which provides an excellent out of the box PaaS-like experience for microservices in the cloud. On the other hand Nomad's more flexible network model more easily supports networks outside of flat cloud topologies whether it's small home raspberrypi clusters or globally distributed edge compute.
There have been lots of speeding, hard-to-catch bullets in the past.
K8s and devops has destroyed us developers' productivity, confidence, and morale.
To me what killed Swarm is Google needing to spread k8s.
If you are eyeing at GCE but decide you want a bit more flexibility, going for a Google backed solution gives you k8s, and Swarm isn’t even on the choice list.
Choosing a third party means you’re on your own to bootstrap/make it live, or contract a vendor that might not live longer than you. That’s fine for companies who strategically choose to invest for the long term, sadly that’s not most of the companies out there.
Kubernetes is for a lot of organisations pretty difficult and way too much for a start. We used rancher for solving our problems which did a fine job, rancher got a worse when they made the move to Kubernetes.
In the beginning they (docker) also removed a lot of options like cpu throttling and configuration and such that LXC had from the start. They also had their own version (a shim) of pid 1, some point at the time. And other things that made it a little painful to properly containerise. I was often very frustrated by docker and fell back on lxc.
There was also something with the management of docker images like pruning and cleaning and other of much needed functionality that just didn't get included.
Also something with the container registries which I a this point can't remember (maybe deleting images or authentication out of the box or something that made it hard to host yourself).
Anyway I think it's failed because it failed to listen to its users and act upon them. They really had a lot of chances I think. I think they just made some wrong business decisions. I always felt they had a strong technical CTO that really was deep on the product but not on the whole pipeline from dev->x->prod workflows.
(Early Rancher employee)
We liked our Cattle orchestration and ease of use of 1.x as much as the next person. Hell, I still like a lot of it better.
But just as this article talks about with Swarm, embracing K8s was absolutely the right move. We were the smallest of at least 4 major choices.
Picking the right horse early enough and making K8s easier to use led us to a successful exit and continued relevance (now in SUSE) instead of a slow painful spiral to irrelevance like Swarm, Mesos, and others and eventual fire-sale.
But if you're on-prem, and have a tonne of metal and just arrived in 2014 via a time machine, swarm was so much simpler and simpler that if you were a sysadmin who already had their own scripts -- their own jenkins-powered IC and git-hooks that built and deployed -- whatever it is you were building -- then swarm looked like a nice gradual extension of that, and k8s looked more like starting over and admitting defeat.
Personally, k8s is one of the easiest system I learned to use in the past few years. It has many components, too many that one can't practically understand all of them well. But every component follows the same design principle, so there's a clear path for a noob like me to get the full picture step by step.
The first thing I read on k8s was controller pattern[0]. Then everything becomes so easy to learn. Something wrong with a pod? Find its controller and start troubleshooting from there. Oh, the pod is controlled by a replicaset? Check replicaset controller. The replicaset is managed by a deployment? You know where to go.
That's why I said it is only complicated in some sense. I saw a lot of people started learning this system by going through a list of popular components and their concepts. I would've gotten lost easily and probably given up if I did the same in the beginning.
[0]: https://kubernetes.io/docs/concepts/architecture/controller/
On the plus side, if someone else configured the cluster correctly for you (e.g. a Cloud service like GKE), then it's a breeze to use.
It was barebones like Mesos but they managed to focus on the right things early on. Once these features and a bit of an ecosystem started to build, the question if we should stay on Mesos with a bunch of homegrown management tooling or move to Kubernetes wasn't even a decision anymore. There was still a significant amount of code which we later just deleted because it became a first class feature or the community implemented something much better.
Does it?
It seems like there are three kinds of complexity involved here: the essential complexity of the total problem set, the essential complexity of a specific company's problems, and the amount of complexity experienced by a user as they try to solve those problems.
Let's stipulate that Kubernetes needs to be that complicated for Google's needs. I'm not totally sure that's true, but let's pretend for now. So we'll say you're right for the first kind.
But I'm still not seeing it for the other two.
For modest deployment efforts, Kubernetes is a giant pain in the ass. It's just way more complicated than needed for many non-Google use cases. I tried setting up Kubernetes for a bunch of personal stuff I run and it was egregious overkill. I persisted until my home lighting system went dark because some internal Kubernetes keys expired and it decided that just not working was a good failure mode. Last time I mentioned this on HN several people said, well duh, obviously you shouldn't run it just for yourself. And that's fine; every technology has an expected operational envelope. But there are an awful lot of projects out there that have between 0.0 and 0.5 sysadmins. For them it's too complex. Kubernetes is just too complex for many people's needs.
And the other side of that is complexity of experience. Modern processors are fantastically complicated. I'm on a mailing list for people in finance who micro-optimize everything to maximize response speed. Those people know incredible amounts about exactly what different processors do. And they have to. But most working developers know about 1% of that because a lot of people work very hard to hide the complexity from them. I don't think that really happened with Kubernetes.
At this point, it is what it is. It's much easier to scale a technology up than down, so I expect it'll always be like this. But For whatever comes after Kubernetes, I'd like to see much more attention paid to more common use cases.
I don't know that if Swarm had those features if it would really catch on like Kubernetes did. They didn't have the same budget as Google on that side of things.
Swarm is still what 99% of companies need when they adopt Kubernetes, and is probably 1/10th to 1/20th the cost. Somebody needs to re-invent swarm as an independent project so that we can stop investing in K8s. The cost of K8s isn't even in the complexity, it's in all the millions of side-effects of the complexity, like security, fragmentation, and specialization.
I guess this strongly depends on one's needs. I mean, if all you want to do is glue together a couple of nodes and have a few stacks running on them, do we really need all the bells and whistles?
I mean, with Docker Swarm anyone can get ingress right from the start on any bare ones cloud provider. Deploy your Docker Swarm cluster, slap a service like Traefik, and you're done. It's a 10min job. What else does Kubernetes brings to the table?
We have about 4300 containers in production where I work (at this moment - it can change from hour to hour), and I don't believe "isolated environments that need different versions of libraries" was ever a driver, or even strong incentive for using containers.
Rather - the ability to rapidly add resources, resize resources, re-allocate storage performance dynamically, and use whichever cloud provider made sense (we've worked with AWS, Google Cloud, and now Azure) - and be able to migrate your environments between them in a period of a week - was what made containerization attractive.
Being able to type:
kubectl scale deploy worker-background --replicas=300
If you've ever worked at a large company - think of the effort required to get 300 new 6 CPU/32 GB VMs allocated to a project you are working on. Now consider a company where pretty much all of the engineering / ops employees can do that from the bash prompt.
I'm just looking forward to the day where we can scale the performance of our storage from 500 IOPS/sec to 8K IOPS/sec dynamically as required (still requires a manual step with Azure - albeit a pretty straightforward one in their portal).
The cycle goes like this:
"A mainframe takes too long to buy, it's too expensive!"
"Here, have an LPAR, you can spin one up in seconds!"
"Wooaa there! You can't just do whatever you want, fill in this paperwork to get an LPAR!"
"Getting an LPAR approved takes too much paperwork!"
"Here, have an Intel server instead! It's not a 'proper' machine, but you can have it in a week!"
"Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new server provisioned properly!"
"Getting a server provisioning approved takes too much paperwork!"
"Here, have a VMware virtual server instead! It's not a 'proper' machine, but you can have it in a day!"
"Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new VM provisioned properly!"
"Getting the VM provisioning approved takes too much paperwork!"
"Here, have a Cloud server instead! It's not a 'proper' machine, but you can have it in minutes!"
"Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new cloud IaaS VM provisioned properly!"
"Getting cloud IaaS VM provisioning approved takes too much paperwork!"
"Here, have a Kubernetes Namespace instead! It's not a 'proper' machine, but you can have it in minutes!" <-- you are here.Containerization's value proposition isn't really the isolation part. That's nice and all, but it's not their main selling point.
The main selling point of containers is the fact that they solve the problems of packaging, deploying applications, and configuration, and they do so in a perfectly auditable and observable way.
I found this passage interesting. Are these tensions common?
It’s quite an American attitude to judge your entire life and skills solely based on the college you went to. I didn’t encounter such attitude with my fellow European and Australian colleagues.
Nowadays, I don’t care that much because I’m not the little, defenseless, graduate student I was, and I do enjoy bringing a Google-Standford-Californian down to earth when they are bragging about something they have no idea about.
Solomon's "outsider" comment is ironic since multiple of us in the room from Docker did, in fact, formerly work at Google and go to Stanford. He walked into that room assuming Google would have those prejudices and that's all he could see.
Amusingly, except the "work at Google"-part, Joe and Craig don't have those other qualifications either. Brendan did get a PhD and was even a professor, but I dunno, he gave up on that.
I also clearly remember the multiple high profile spats that 'shykes had on HN which burned a lot of bridges. At the time he had a reputation for answering lots of questions on HN which helped a lot with community building. After those bridges were burned there was no one to speak for Docker as the developer mindsets shifted slowly towards Kubernetes. IIRC the Github PRs were also a source of contention as dotCloud corporatised.
To be fair, Kubernetes was a real slog to understand at the start and had a lot of competition; it was definitely not the same level of simple, direct technical solution that Docker was.
Interesting trip down memory lane and what a pivotal technology! Regardless of the rest Docker is a true cultural phenomenon and a testament to the insight of the creators working outside of the myopia of big tech.
Speaking personally, the fate of Docker, Inc. was clear to me when they took their $40M Series C round in 2014. I had met with Solomon in April 2014 (after their $15M Series B) and tried to tell him what I had learned at Joyent: that raising a ton of money without having a concrete and repeatable business would almost inevitably lead to poor decision making. I could see that I was being too abstract, so I distilled it -- and I more or less begged him to not take any more money. (Sadly, Silicon Valley's superlative, must-watch "Sand Hill Shuffle" episode[1] would not air until 2015, or I would have pointed him to it.) When they took the $40M round -- which was an absolutely outrageous amount of capital to take into a company that didn't even have a conception of what they would possibly sell -- the future was clear to me. I wasn't at all surprised when the SVP washouts from the likes of VMware and IBM landed at Docker -- though still somehow disappointed when they behaved predictably, accelerating the demise of the company. May Docker, Inc. at least become a business school case study to warn future generations of an avoidable fate!
[0] https://news.ycombinator.com/item?id=8682525
[1] https://www.hbo.com/silicon-valley/season-02/1-sand-hill-shu...
Without a clear idea of what specifically you need to spend money on, and how specifically you are going to spend that money to generate growth and value, taking money is a bad idea. And if the answer to that question is "so that we can keep paying our developers", then you're probably already doomed.
Docker went to war with the standard Linux init system and the kernel developers who maintained cgroups, their core system primitive. I don't really understand how that was supposed to work out.
This for me is one of those "I feel like I'm taking crazy pills" lessons. It seems so obvious to me, and I hesitate to even estimate the vast amount of cash wasted in the last decade like this. But it's a lesson that an awful lot of people seem to insist on learning for themselves. Or not!
> Craig McLuckie, Kubernetes cofounder and now vice president at VMware, says he offered to donate Kubernetes to Docker, but the two sides couldn’t come to an agreement. “There was a mutual element of hubris there, from them that we didn’t understand developer experience, but the reciprocal feeling was these young upstarts really don’t understand distributed systems management,” he told InfoWorld.
The article criticizes Docker Swarm as myopic, but IMO, there were only two possibilities for Docker to move forward; either they acquired Kubernetes, which was a possibility in this telling of events, or they won with their own Docker Swarm.
I am wondering if anyone ever actually properly used docker swarm in any proper way? Especially as a small-scale developer.
Swarm was good, easy to set up and run, but it had a lot of networking bugs. More than once we had to cycle the docker daemon on all machines which basically resulted in a rolling outage.
We’re trialling nomad at our new co, I was put off by the opaqueness and complexity of K8s, but apparently that’s just me.
At least, the founders of Docker are honest and clear in my opinion:
<< Hykes disagrees with this assessment. “I think that is wrong and generally speaking the core open source product created massive growth which created the opportunity to monetize in the first place,” he said. “Lots of companies monetize Docker successfully, just not Docker. There was plenty to monetize, just Docker failed to execute on monetizing it.>>
There is still a thing that is missing there: the article let think that docker was an innovation out of nowhere and they had an unique idea that was kind of spoiled.
But, one has to remember the context of the period when Docker was created:
It was a time were cgroups and "namespaces" of network, process, ... were the hot new things inside the Linux and everyone was thinking about the concept of "container", one way or another.
There were already chroots and a lot of persons were already working on "app"/"module"/"package" systems using aufs/overlaysfs layers.
So, this was the moment and a few competitor were emerging like lxc and docker. Docker was very good to take the of the media cover and hype and so to become the dominant way to have "containers". But, at the beginning, there was nothing particular in the technology and lots of competitors could have done it if it was not Docker.
Really, Docker's innovation wasn't the ability to run processes in a container (as mentioned before, that technology had existed for years already); it was the Dockerfile and the Docker hub/container registry. Prior to Docker, the way you'd build a container was by running debootstrap or unpacking a tarball/zip file of a root image. Take a look at https://developer.ibm.com/tutorials/l-lxc-containers/ . As far as I recall, there was no LXC equivalent to `docker build` or the registry except maybe wiring up Packer or home-grown shell script and throwing a tarball in S3 or something.
What made Docker really explode was when it ran on Mac and Windows. The cloud runs on Linux, but desktop computers generally don't, mostly for cultural reasons.
This made for an explosive combination. There are probably similarities to Github here.
> Red Hat specifically “weren’t part of the community, they never rooted for the success of the Docker,” he said. “The mistake on our end was desperately wanting them to be part of the community. In retrospect we would never have benefited from that partnership.”
I understand how it seemed like other open source companies were dictating terms, but Docker's implementation was crap: one fat daemon running as root, no sandboxing, no cryptographic signatures, and a total lack of integration with the rest of Linux. Red Hat gets to define how containers should work on the back end because it pays for ~7% of all kernel development (compare that to Canonical's ~1%) [0]. But instead of cooperating with others in the Linux ecosystem, Docker only begrudgingly adopted interoperable standards while Red Hat, CoreOS, Intel, and others implemented fundamentally better solutions.
> We didn’t work at Google, we didn’t go to Stanford, we didn’t have a PhD in computer science. Some people felt like it was theirs to do, so there was a battle of egos. ... Kubernetes was so early and one of dozens and we didn’t magically guess that it would dominate.
Google had experience running containers at scale and was using K8s as an opportunity to retool their internal proprietary solution. It sounds to me that Google engineers tried to explain to less experienced team the best path forward ... and Docker took it as an insult.
Sure, not everyone needs all the features of K8s, but Docker could have built a simpler interface on top of K8s and benefited from Google's investment. Instead they built a half-baked competitor that didn't work well. Whoops!
Docker delivered a slick developer UX but their architecture was antithetical to how Linux developers thought it should work. Then they spurned offers of help from both the hyperscalers and the dominate Linux enterprise vendor [1] and chose to build inferior solutions for those markets. Now they are stuck selling their pretty UI because that's the only thing Docker does fundamentally better than anyone else.
[0]: https://www.linuxfoundation.org/wp-content/uploads/linux-ker...
[1]: https://www.redhat.com/en/blog/red-hat-continues-lead-linux-....
What are you waiting for, is there something new in the works wrt swarm? I was a swarm fan early on(and secretly still am), but otherwise k8s is where I spend all my time at work.
Of course when it comes to distributed environments I'm sure Kubernetes still reigns supreme, but for relatively simple container setups I don't think there's a need to even run Docker anymore on Linux.
Obviously, I'm still not well versed in containers so this might all be overlooking something.
If I were at the helm of Docker today, I'd focus on Enterprise customers. Provide an on-prem version of DockerHub and you'll convince thousands of companies tired of Artifactory to switch immediately. Don't get me wrong Artifactory is pretty nice, but it's a bear to run at scale on an enterprise level. DockerHub already is handling that traffic, so there's no reason to think they can't port that to an on-prem offering.
It's too bad they didn't make the business side of things work, though I agree that at the time there was a certain feeling of too much money chasing an uncertain future for an open source project. I hope this next iteration of the company works out for them.
The only changes that they really rolled out were replacing the older, perfectly functional UI with something superficially shinier but much, much harder to use.
Not only was it just hard to see what you wanted to see once they'd halved the information density and got rid of any visual hierarchy, you had absolutely crazy things like a slider for the number of instances of a service that would instantly apply with no confirmation and, even crazier, responded to the scroll wheel. We once doubled our number of production pods and didn't notice for a few hours, because someone scrolled up the page and their cursor went over the slider. Lucky they weren't scrolling the other way.
Anyway, I lost a lot of trust in Docker as an organisation that knows what people value about their products as a part of that.
Unfortunately they couldn't make docker into a product (or they would have faced the wrath of the OSS community) so they probably felt stripped for time on both docker and on swarm, which made for an unpolished experience.
I've been using swarm for a small project for funz and because I didn't want to run the full k8s (even k0s or k3s would have been heavy for my use case) and because I had bad experiences with nomad.
My list of complaints: - Stacks are not working 1-1 with Compose - docker machine is a separate binary - You can't pull the latest of a docker image or the cache hit will just fail to redeploy your changes (you're forced to do a sort of blue-green deployment or just update a tag everytime) - You have to configure, run, garbage collect a registry from scratch - docker-flow-proxy should be included out of the box - I need a way to integrate secrets
Don't get me wrong, there's an amazing amount of progress in docker swarm (and I'm running it in production, for free, on a 5$ machine). Years ago it was even worse but it's not something I would ever recommend to my employer. I hope we'll get there eventually.
Docker, the kabillion dollar company, never had a chance. We all recognize the potential and actual problems when some very crucial piece of infrastructure is only free/open source and not maintained -- but this is what happens when it is "overmaintained" I.e. it makes sense to invest big in something that is good and could be crucial; but only if you can both keep up quality AND stave off competitors -- and in this case, competitors aren't just other companies, but free software in general. It's kind of funny, in a way, Docker made itself a target for "good competition" by being so visible, in a way that e.g. "curl" didn't.
Either way, I always knew "Docker the kabillion dollar company" was a stupid stupid bet. Small company, great bet. But this? Destined to fail.
Anyway they are past the point of no return now. Their creations will undoubtedly outlive them - probably Queen Elizabeth also. Anything they pull back will be replaced by the community. Anything new will get aped by everyone else trying to make a name for themselves. Containers are more or less a generic commodity now.
Heck, I use plain old docker-compose to manage production servers with half a dozen containers, trouble-free for a few years now. Compose v2 seems promising, I've been using it for a while and it's pretty stable too.
Swarm could have been an industry in itself, a de-facto standard for orchestration if it had been marketed and supported right. It was a lot more stable than Kubernetes in the early days and things just worked with docker while you'd have to kludge a few things to get Kubernetes doing its thing right.
I actually got an AWS ad on the same page while reading this comment. It's definitely hard to compete when your competitors(with lot of capital) can easily integrate your tech into theirs in no time.
if they had just released a chroot database runner, would have been as good as what they did
it has never been good at build or deployment. it has definitely never been good at cloud. even kube is still awful at most things
these systems have never been sure if they're configuration languages, buildsystems, plugin hosts, or operating systems, and so they've been bad at all 4
Programmers hate things that are easy to use (by the way I am a programmer, and no I don't feel that way) because if they admit they use them then they might be accused of being users. And of course no programmer will admit this. They don't realize it because it's actually a subconscious psychological issue.
And so what programmers started to do was immediately make it much more complicated and at the same time, take it completely for granted. It was so useful it was like a floor to walk on. And so people started giving it the same level of respect they give a floor.
So for those reasons, Docker became very uncool. But at the same time it was incredibly useful. Solution: make something just like Docker but not Docker, which hip people will be allowed to use without any shame. Make it a bit more complicated and only run on the cool expensive hardware.