So nevermind a retort, what I would like to see is a sane, sensible "business value" cost/benefit, pros v. cons breakdown of just what the heck you're actually gaining (and losing) using Docker vs. some other architecture/methodology. Because absent that it's all just hype and kool-aid drinking in my opinion.
What would help with the above is if people would document what they are doing with Docker that works, because either they are hurting but not realizing it, or the author of the article is just "doing it worng" and whining about it in public. What is really going on with Docker, et. al.!?
We had some issues with docker-compose for our use-cases, such as persistent database even when recreating every container from scratch, so I hacked on some stuff that become "dup" [0] (nee. docker-up)
Ps. it's written in Nim, which was a lot of fun!
Docker is a slow, blunt tool for just containerizing dependencies IMO.
Always beware of perception bias or whatever the right term is for this. When docker was new it got tons of hype and people were talking about how great it was. Now it's no longer the hot new thing so it's just another tool many people are using quietly and successfully. It no longer warrants much positive discussion. The people using docker in production successfully don't feel compelled to write articles about it working for them.
I'd say the pros/cons of docker (or containers in general) are similar to the pros/cons of buying hand made vs. mass produced goods. The hand made goods will probably be higher quality but they take more time and skill to produce so they're more expensive. The mass produced goods will probably be lower quality but will work fine and at least offer consistent/reliable level of quality. They take less time and skill to produce so they'll also be less expensive. In the context of using Docker in production this translates to it being very good for solving 'off the shelf' types of problems quickly and reliably. It's essentially the software/systems version of an assembly line. If you don't need an assembly line then docker may be worthless or even a total waste of time. If you do need an assembly line it's the best option currently.
There aren't articles about docker in production because barely anyone adopted it for major production environments [yet]. And for the few who are early adopters, there wasn't enough time to write articles.
* Docker encourages fully disposable infrastructure
* Docker containers can be more secure then traditional environments
* Docker provides for indempotent environments agnostic of hardware concerns etc
* Container management software (Kubernetes, etc.) makes Docker much more powerful/useful and is only going to get better
Specifically, I'm curious if my observations are correct, and if they are why they are correct and why are they SO much better (vs current docker alternatives including non-container based approaches).
>>> * Docker encourages fully disposable infrastructure => I'd consider that incorrect and/or misleading
Docker is meant to manage disposable (i.e. stateless) applications. They are expected to be killed/stopped/restarted at any time, no persistent state, no data.
An application is either disposable OR not disposable by its design, docker doesn't change that. (For instance, don't try to dockerize databases).
Docker gives you a quick, nice and easy way to package your applications with dependencies (as Docker image) AND deploy them. [This is valuable]
It's especially great for development and test environments because you can run many containers (i.e. applications) on a dev/test machine, without dependency/deployment hell. [This is very valuable]
>>> * Docker containers can be more secure then traditional environments => No. Just No.
Docker has nothing to do with security.
>>> * Docker provides for idempotent environments agnostic of hardware concerns etc
Docker gets to ignore the hardware. You do not.
You get to buy, setup and manage servers. Maintain a full environment that can run docker.
It can be an undertaking full of dangers, as told by this article and mine :D
>>> * Container management software (Kubernetes, etc.) makes Docker much more powerful/useful and is only going to get better
Docker allows to package and deploy an application. It doesn't do anything about infrastructure at all (i.e. machines, load balancers, DNS, discovery), which you still have to manage and it's a HARD problem.
The orchestration systems is supposed to help you manage all of that. As a result, it will be able to automatically kill/restart/move/add containers across a cluster and make the necessary [physical or virtual] infrastructure changes.
Container + Orchestration will be a [r]evolution to manage entire giant fleets of hosts and applications. [This will be infinitely valuable (allowing me to automate the work of 10 people, all freeing all my time to enjoy unlimited free drinks and foods)]
IMO: In the current state of affairs. Having docker without orchestration is like having cars without roads.
Firstly, I think it is correct that docker encourages disposable infrastructure, because your docker setup basically must be able to recover from the loss of a container. I think it would not be sane to expect your containers to "just keep working".
Regarding security, I don't think Docker provides much other than the illusion of better security at the moment. At least on Linux containers don't add security that you can't get with traditional services.
I suppose if you use authenticated images you can at least be sure that you're running the code you think you are, but then again package managers have had signing for ages.
You're always going to have to test your software in the environment you will be deploying it in, because driver bugs and weird interactions still exist. Docker doesn't really let you ignore the hardware
I have no opinion of kubernetes and such at the moment.
What docker does grant is an easy method of "packaging" software... Pretty much anyone can just throw everything together in a big bundle and distribute that. For certain use-cases, this can give a development team much more speed, since setting up a workable environment is much faster. There is also a nontrivial amount of software that simply does not bother with OS packaging where containers can help you deploy it in a more controlled fashion.
For sure for developer it's a big thing, for SysAdmin and people thinking about the datas it's more a pain in the ass.
Anyway, no way to go back. This is the future Jack.
On the last DockerConf you could ear ADP talking about their success with Docker, but obviously their there to promote Docker. ;)
Absolutely. You will get the most out of it like this. You can run pets, and treat containers VMs but I don't think you will get all the advantages.
* Docker containers can be more secure then traditional environments
Out of the box you get isolation from namespaces (not being able to see the host filesystem or network), resource limits via cgroups, a default seccomp profile that restricts some system calls that could be used in exploits, and more. You can set all these defaults yourself but the majorty of people do not. So there are definit benefits.
* Docker provides for indempotent environments agnostic of hardware concerns etc
Most applications do not talk directly to hardware, the default container does not directly present hardware interfaces, yes.
* Container management software (Kubernetes, etc.) makes Docker much more powerful/useful and is only going to get better
It does help to have software to manage programs across multiple computers, for scaling and redundancy, yes.
* Use modern tools to spin up a dev env * Determine how to monitor and maintain dev and deploy * Test dev env * Determine, based on dev and a deep risk assessment how prod will look * Create prod env * Determine how to monitor and maintain prod env and deploy (based on your test) * Test new prod env * Open up prod env to end users
It's not bloody rocket science - it's Engineering and the steps above hide loads of steps and feedback loops such as a safe upgrade method eg snapshot/clone VM, test, release.
If your product is important to you then you take it seriously and treat it right!
Cheers Jon
PS Did I mention documentation?
I keep reading articles stating that "the Docker API changes with every release", but the assertion is never backed by any specific examples. Has anyone here encountered an actual breaking change? If so, I would appreciate you sharing the specifics so we can fix it.
Docker is by no means perfect:
- I remember that in 1.10 the switch to content-addressed registries meant that older clients could not pull by digest (but all other commands, and even non-pinned pull, still worked). This was not an accidental breaking change: it was the result of a difficult tradeoff. In the end we decided that the benefits of the new content-addressed model outweighed the inconvenience. To guide our decision we used data from Docker Hub to assess how many clients would be affected. I forget the exact number but it was a very small minority.
- And in 1.12 we got bitten by a change in how Go 1.6 processes HTTP headers (it became more strict and thus rejected headers from older clients). That was quite simply a screwup on our part.
So we've had our share of screw-ups, no question. But lately I've been reading the "breaks at every release" meme more and more. Based on the evidence I have, it seems incredibly disconnected from reality.
What am I missing?
http://stackoverflow.com/questions/37617400/ sums it up. There's a magic (afaik undocumented) env var DOCKER_APIVERSION you can set for compatibility, but nobody can find it.
If you can't mix even minor versions then yes, it's a problem. I can't run X in prod and anything other than X in dev of I'm working with the same toolset.
Thanks for bringing this issue up though, because we are currently working on a fix. Here is the PR: https://github.com/docker/docker/pull/27745
As for DOCKER_API_VERSION, it is possibly not well documented, but it is indeed present in the documentation: https://docs.docker.com/engine/reference/commandline/cli/
Hopefully this was helpful.
After verifying, I have to rectify your second point: 1.12 DOES NOT BREAK older clients.
To add more information, the issue was present for Docker clients with version 1.9 and older talking to THEN Docker master. Thankfully it was seen before 1.11 was released, and thus we decided to stay on Go 1.5 for Docker 1.11 for this reason.
For 1.12, we added a workaround for this issue, allowing us to upgrade to Go 1.6.
UPDATE: Docker 1.10 was built with Go 1.5 so this issue was not present with a Docker 1.9 client talking to a 1.10 daemon either.
I don't know how incorrect I am, but it seems like the Docker project is more interested in features rather than stability. Except, like you said, I don't have any specific examples to back that up. Maybe it's just community hive mind thought bleeding onto my decision making abilities?
The most concrete failing I've experienced was the deprecation of the boot2docker project in favor of docker machine. When I started using docker machine, stuff just... broke... and it left a bad taste in my mouth. In fact, using docker on either a mac or windows is just awful for me. My anecdotal experience is that it breaks in small ways that then that can really disrupt my productivity. I'm in the process of picking up docker again and it led me to install Arch on my work laptop just so I can have native containers.
Right now my use case is using containers to test Chef cookbooks and it works... alright. Not as well as virtual machines, but alright. How docker handles init bothers me a little bit but it's nothing the maintainers of the testing suites for Chef couldn't handle through yaml configuration options.
Oh, I guess while you're here... one thing I've been wrestling over is the usage of dockerfiles. I have no problem with containers but to me, it looks like using a configuration management system to bake container images seems to grant so much more flexibility and testing capabilities that I can't really see dockerfiles as anything other than an intermediate step. Is this a valid viewpoint?
I think that is a fair dig but I think they are still exploring the limits of what it can do. They are following the path of release often and make changes and show your working really well - you get to see the code and can contribute yourself.
The thing is that you have to take all that into account when you are using their product. You need to take an Engineering approach to "production". Use Docker in the way that works for you. You state that you ended up using Arch instead of Windows or Mac - so fine, you have constructed a tool chain that works for you and welcome aboard! You see, Windows and Mac also do odd things every now and then but you don't get to see the workings.
You are clearly not daft - you use Arch. I run Arch (my wife's laptop), Gentoo (my laptop and a fair few mail proxies and Icinga instances and other things), a lot of Ubuntus and a slack handful of Kalis and a sprinkling of Debians, Centos and others.
You need to use engineering principles when going to production. Also you have to look at the price you paid to use Docker - it's a purely social one but one that is extremely important and may not involve money but cost you a fortune anyway. You need to decide if you want to pay that cost.
You are clearly someone who understands the open source methodology but I would suggest that you look a little deeper into the dev -> prod thing. There is no reason why dev = Docker => prod = Docker. That's something that only you can decide after careful testing.
Cheers Jon
I personally find Docker a phenomenal resource for development. For production however, you analyze your use case, the risks, the failure modes etc etc and then you decide whether Docker is a part of that.
I'm by no means a massive Docker user but compared to many of the sub systems running on my laptop it has been very stable for me although I do run Gentoo. Actually my entire laptop is surprisingly stable TBH. I absolutely love the way I run a few short commands and a fully tooled up web app with all the supporting infrastructure simply appears.
Funnily enough, only yesterday I had a look at this: https://nextcloud.com/collaboraonline/ which I found in an article somewhere (LWN?) Yep - love it (NC) so the next step will be to decide how to use it for my home production system (yes I have dev/production at home) For that I will use pretty much the same criteria that I will use for my office and finally my customer systems. In this case Docker for production probably makes sense but I need to analyze it a lot more. A CoreOS VM will probably be spun up very soon in my attic for stage 2 in home production.
I really don't like to see rants along the lines of the other article, denigrating a product that a lot of people have worked hard on, that shows a lack of discipline and clarity of thought. I for one would like to thank you for your work and making it available for all. It has genuinely saved me time and effort and made my IT life a little bit better.
To answer your final question: a thicker skin, mate 8)
Cheers Jon
I created an account here so I could let you know I've clarified some points in my post to address your concerns.
The fact that if your local client is newer than the daemon, the client will fail to do anything is pretty annoying. You can hack around it, but that's not a solution to the problem.
Also, while you shake off the 1.10 change as though it wasn't a huge deal, the migration tool that was written was not fit for purpose. First of all, it sometimes caused corruption after the migration (fixed after release but still an issue, and you couldn't practically un-bork the corruption). There were other issues with the tool, but the biggest issue was the there wasn't any sane transition period -- which made packaging very difficult. Because Docker couldn't handle restarting the daemon and keeping the containers around (and after restarting a post-1.10 daemon you'd have to wait for the full migration to finish before you could even interact with the daemon -- another issue with the transition), you'd have to publish a fake version bump of the Docker package to even alert users to the issue!
A more sane way to handle it would've been how databases handle the issue. You lazily do the conversion (or you have your software be able to handle both formats so you can run your migration while still having your containers work). And yes, I'm aware that it'd be difficult to implement but the transition path implemented was really bad.
I spent several weeks of my life (that I'll never get back) trying to make the migration path work. It didn't end up working because it wasn't designed to work nicely from the outset. So I ended up just telling users of our Docker packages about the issue and packaging the migration tool separately.
Another issue that I've had recently is that Docker is making it increasingly difficult to use `runc` by another name. In openSUSE we package a system runC separately to Docker (and it lives in /usr/sbin/runc), but because of the fact that in 1.12 --runtime was intentionally broken (and containerd's defaults are no longer obeyed by Docker) now you can't effectively use --exec-opt native.cgroupdriver=systemd. Great. So now we're going to have to package docker-runc and docker-containerd separately.
Don't get me started on the containerd split. I was not very happy the day that PR was merged, because it caused a bunch of headaches for us. It's also a bit amusing to note that containerd was tagged as "pre-alpha" until Docker bundled it (then it was considered stable).
That all being said, Docker is a great project -- it's just that you have to admit that issues with Docker are not "incredibly disconnected from reality".
I think it either came down to a broken version of docker at the time that couldn't run properly.
Or it was because of unstability from Docker. All our Erlang apps are meant to take thousands of requests per second and Erlang have particular ideas about how the system/network should behave. It's very hard on Docker and any unstability or incompatibly would make up for outstanding failures.
Either way, we know for sure now that there were many major critical issues with Docker at the time of the Erlang trial. Not adopting it was the right decision, and it still is.
We only use Erlang for critical applications. We really can't afford to push the limits after a few red flags.
That says all you need to know about the trustworthiness of Docker.
EVEN DOCKER PROPONENTS caution against using it in 'important' apps....
What apps are people investing time in that aren't 'important'?
Is there a coffee machine that is ok to use for a docker app somewhere?
Most coffee machines are docker ready.
To guarantee you the best experience, you will need to setup a pair of coffee machines, plus an orchestration system that will be responsible for swapping them automatically when one ran out of coffee.
Note: There are only prototype of orchestration systems. Nothing for sale in the corner shop yet.
---
More seriously...
Not important: Most internal, development, and test systems
Somewhat important: Web applications, various support micro services. (They all are stateless, with multiple instances, and reactive failover by their respective load balancers).
Critical: Most databases (especially the ones without multi-master mode and automatic failover), trading applications, payment systems, accounting systems, databases with money $$$
This whole ordeal has actually been a really really good lesson for me about mature vs less than mature apps. It's been really enlightening to read the posts and discussion.
Docker is certainly having some teething issues - and things like kubernetes are certainly WIP. But I wouldn't be as skeptical - plenty of companies use it in production already, for "core" apps.
Having just come off a year at large(very large) government department that have 100s of core services running in Docker in production, I think the tech is ready, but not sure if every company is ready.
Sorry, but this is really a bad advice. We have ran and contine to run various databases inside Docker including MySQL, PostgRedis, Cassandra, Elastic Search, RethinkDB even HDFS with proper user rights and configuration. We can maintain the state just as fine. If your only problem is to move the data, all you have to do is stop, export, tar it, move to another server, just as you would do in a normal server. Docker is not a magic bullet to solve such kind of issues. Yes, Docker might have another problems, but just as you could not run someting with state inside Docker does not mean "thou shalt not run" , there are various ways to manage state. Host, IO can get crash regardless of Docker.
Docker and the like seem like a solution in search of a problem for this particular part of a typical service infrastructure.
Or, to put it more bluntly, just because you've gotten away with it (thus far) doesn't make it a good idea.
What is your typical server infrastructure? We use bare machines and had kernel panics very rarely (only with experimental MACVLAN). We did not 'get away' with anything. We have been using Docker for over 2 years, and LXC much more and know how to handle mounted directories, it is not a rocket science. We used LXC, now Docker to provide quick isolated mini linux environments. They are just different type of Virtual Machines when you abstract it in your head, and do not expect it to do magical stuff like saving your data and send it to another host upon crash.
> Native storage software packages are available for mainstream OSes that handle dependencies via the native package manager.
They are often outdated, we mostly get one of the latest builts, and in some cases Snapshot builds, where they are not available anywhere. So ability to run versioned instances of some software independently and isolated is a huge win for us. We do not want to distrub people using the old version. Yes it can be solved with VMs. We use our own bare metal servers. We do not like to "pay" to open VMs, so we turned to OpenStack. It is very complicated for our use case, we like to use bare metal on our own. But, we still need isolation, this is where containers; LXC and Docker came to help and saved us a burden.
Put your database in a container, let it store its data on a network storage system (like a distributed file system). The container scheduler makes sure it gets rescheduled when the node goes down.
I've been using Docker in production for a single server website and have had very few issues. I do like how easy it is to reproduce a working environnement with a "docker build" though.
That being said, I think that just using Ansible on a server is probably an easier and more reliable solution. Ansible is battle tested and allows to have reproducible environments too.
There is also work in progress:
While there were teething issues, this article does a good job of pointing out the flaws in the original article, I think. It's been easier to get our team up to speed on Docker and it's gotchas than nearly any other configuration management, server management, et al. systems that we tried!
Long version: I met the DevOps guy who [I believe] is responsible for pushing Docker at Expedia and we've had long conversations about it.
They were lucky to have had a particular environment and a specific version that worked, and got it pinned down and frozen very early.
I suppose you are on the dev side and not aware of all that. (Hell, maybe, you're not even in the same subsidiary of Expedia). I'm glad it all worked out for you as a dev, my devs are also happy with Docker. (We're probably used as an example of Docker success story at times).
In the end, there is no free lunch. There is dirty work done and more to be done. Some of which is invisible.
We're an external team (completely seperate to Expedia) that is brought in to build certain products :)
And yeah, definitely. I spent a couple weeks debugging the ECS issues we had with the Wotif guys, which was fun, but none of the issues were insurmountable!
We were one of the first adopters for the new ECS deployment, and while some parts of it weren't fun (Splunk still drives me mental) for the most part it went smoothly: that's down to how good the ops team is, I think.
Our specific case was somewhat special, in that because we're outside of the rest of the infrastructure, and had extensive experience with Docker in production for other clients, our apps were basically ready to rock from the get-go. I think we were the first PHP deployment within the new ECS deployments!
Should send me an email, it's in my profile, would love to chat sometime!
I've never seen this limitation with other container solutions. What is it about Docker that makes it problematic?
The thing I've learnt with Docker is that all the other Prod issues still exist. Docker at its core only solves the executable part or distribution of the process to run. Still need to figure out, network, storage, monitoring, backups, discovery, etc...
What starts out as a single host can quickly quadruple once all the other considerations are taken into account and wanting a scalable, reliable and available system.
A Docker registry has own authentication system. So does AWS (and GCloud). So what you end up is one wrapping the other: To access the ECR, you have to run an AWS command to get a token to put into "docker login". Google has "gcloud docker login" for the same purposes. Both produce temporary credentials that time out, so can't be used for long-running things.
This means that any tool designed to work with a Docker registry needs to support this particular workflow. For example, this affects Drone [1].
It also adds complexity. GCloud is particularly heavy on the authentication complexity side already (compared to AWS's comparatively simple keypair approach), and with SSH, GCR and Kubernetes on top it starts to stack up in ways which can make users' head spin.
Straight Docker Hub is refreshingly straightforward by comparison.
[1] http://readme.drone.io (not to be confused with drone.io)
[1] https://github.com/GoogleCloudPlatform/docker-credential-gcr
[2] https://cloud.google.com/container-registry/docs/advanced-au...
This article: "It'll be better in the future, you'll see!"
The former is verifiable, the latter is a hypothesis.
If you really want to impress your interviewers, back up your criticism with a pull requests fixing the issue. That will automatically put you on top of the pile of resumes.
Typo: a wash => awash
I know! The content is more important than the quality of the writing, but it's a little surprising to see such a mistake jumping at the reader at the start of an article. We should go back to the first days of the Internet where "updates" were possible. :) I would have loved to suggest an update quickly.
And it will still be valid if someone forks Docker. In fact, that would validate the criticism.
