Some time ago I've written about my experiences using it in production https://nickjanetakis.com/blog/why-i-like-using-docker-compo.... Not just for my own projects but for $500 million dollar companies and more.
We have been using it for more than five years now. Before that, we had a legacy deployment model, and I do not remember a single major issue related to Docker Compose.
We use it for both staging and production environments. The same Docker image validated in staging is deployed to production. Never fails!
This is why nobody uses it. Cloud stuff has to be as baroque as possible.
https://github.com/daitangio/misterio
It works very well!
But docker compose can actually be very sufficient for what many projects actually need.
Granted I am a guy pushing for compose based localdevs and such but going further you often just cannot beat the simplicity of doing update QA or other CI/CD workloads in compose based projects. I have had dozens of projects where we replaced flaky slow and maintenance heavy pipelines with just docker compose up --build --wait in the past years. How come you say health checks are still broken?
Agree.
Plus there's the monitoring of the host that is always overlooked in articles. I've ended up chucking Monit on there to monitor disk usage et al, and also used it for monitoring compose too and restarting containers.
And then there's Healthchecks.io, and external uptime monitoring... the list goes on. Properly monitoring systems, even single server systems, is not simple.
While not built in k8s has at least velero and kasten. However they are only possible because of snapshots https://kubernetes.io/docs/concepts/storage/volume-snapshots... and kasten has a plugin like architecture (because of k8s ) that supports application specific backups. However I never found something like that for compose. And that is troublesome in bigger projects like sentry
Docker volumes (and bind mounts) however have the minor problem of being hard to get a consistent copy to without stopping the service. You can work around this by, e. G., having ZFS or btrfs as the underlying FS and making a snapshot there. Otherwise, your software (like PostgreSQL) might also have other online backup tooling.
It's because I like keeping my servers stateless when possible. It makes it easier to upgrade them in a zero downtime way later.
If your web server has your DB too, then you can't do zero downtime system upgrades. For example I would never upgrade Debian 12 to 13 on a live server. Instead, I'd make a new server with 13, get it all ready to go and tested and then when I'm ready flip over DNS or a floating IP address to the new server. This pattern works because both the old and new server can be writing to a database on a different server.
With all that said, if you were ok with 1 server, then yeah I'd for sure run it in Docker Compose.
For most DBs it's one or two paths in the container, and virtually all DBs vendors have a reference Docker Compose example somewhere showing volume config. I can't remember the last time I ever "natively" installed a DB personally!
It really comes down to YMMV... Sometimes for a singular app surface, it's easier to just use a compose file that includes the database. mailu/mailcow is a good example... you don't necessarily want to comingle email on the same server as other services.
That said, if you need to share a single DB or set of DBs across an applicaiton with several instances/deployments, then it makes much more sense for a central deployment. I almost never do my own host level install, instead relying on cloud hosting and mgt. The only real exception is MS-SQL on internal servers... MS-SQL in Docker is barely acceptable for dev, and missing a few key features you may actually want/need.
By the way most docker based implementations do not actually need the userland proxy docker runs automatically. Disable it in /etc/docker/daemon.js
{
"userland-proxy": false
}Like, if that works for you, more power to you. But that is a lot of moving parts in exchange for using a tool whose value prop is that it doesn't have many.
Also you don't even need the loopback address if the traffic is between one container and another, just a bridge network is fine.
For forward chain we set docker network ranges to route between themselves and only services actually used in containers. Allow container outgoing connections to our DNS servers, centralized HTTP proxy server and monitoring - nothing else containers are allowed to route to.
And for output is similar, only allow our DNS servers, NTP, HTTP proxy, centralized rsyslog where everything goes and zabbix monitoring server and a few icmp types - nothing else gets out and is logged.
With the advent of these supply chain attacks we read about often here it's just a matter of time some container is compromised and this seems like only viable way to at least somehow limit impact when such an event occurs.
The only modification is that I pin containers to an IPv4 address so I can limit the forward rule to that address.
What if you can't by yourself objectively evaluate if turkey sandwich sounds good?
It's not a matter of giving a universal answer to whether docker compose in production is fine, but how to evaluate it. Which features or safeguards necessary for a healthy production environment you forfeit when choosing plain docker compose? What's the tradeoff?
Comments like this are apathetic and reduce the challenges of good software engineering to hopes and random chance.
More to the point, there is no objectively right answer of what stack you should use. There are plenty of objectively wrong answers, but compose isn't one of them.
*shudder*
There’s a reason articles like this exist. Things change.
journald will help with logs, and the pull policy[1] helps with mutable tags. What help do you need with "orphan containers"?
[0]: https://docs.podman.io/en/latest/markdown/podman-quadlet.1.h...
[1]: https://docs.podman.io/en/latest/markdown/podman-image.unit....
You shouldn't be using podman compose. It's flimsy and doesn't work very well (at least it was last time I used it prior to Podman v3), and I'm pretty sure it doesn't have Red Hat's direct support.
Instead, activate Podman's Docker API compatibility socket, and simply set your `DOCKER_HOST` env var to that socket, and from there you can use your general docker client commands such as `docker`, `docker compose` and anything else that uses the Docker API. There are very few things that don't work with this, and the few things that don't are advanced setups.
For what it's worth, podman has also a thin wrapper (podman compose) which executes `docker-compose` or the old `podman-compose`. The docs should explain which it picks.
Note:
- `podman-compose` is an early attempt at remaking `docker-compose` v1 but for Podman. This used parsed the compose config and converts them to podman commands, and executes it.
- Later Podman wrote a Docker compatible socket instead, which can work with most docker clis that accept a `DOCKER_HOST` argument, including `docker` and `docker-compose` (both v1 and v2)
- `podman compose` is a thin wrapper that automatically selects `docker-compose` or `podman-compose` depending on which is installed.
Generally all you need is podman, docker-compose (the v2 binary), and that's it. From there you can use `podman` and/or `podman compose`.
Moreover, podman-compose integrates really easily with systemd. You can create a service by just running "podman-compose systemd" and following the prompt! No quadlet nonsense required.
https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
Having your whole application with its containers, volumes, and networks all defined together in one easy-to-read YAML file is a way better experience. Deployment is two steps: 1. `git clone foo` 2. `docker compose up -d`. You can see the state of the application containers with `docker compose ps`. You can run multiple compose applications on the same host and manage them separately by putting them in different directories.
With quadlets, you delegate everything to systemd. You have to break the configuration up into a bunch of tiny unit files and then separately copy them to /etc or a dedicated user's dotfiles. An application with a handful of containers and multiple networks/volumes/etc can spiral into a dozen unit files. Good luck SSH'ing into an unfamiliar system and understanding at a glance what it's doing. It is far more annoying to predictably deploy and tightly couples your application configuration to the host system configuration. (Even moreso if you created dedicated users for each application, which I understand is the recommended solution.)
If I'm just holding it wrong and there exists some better tooling to manage podman in prod that I don't know about, I'm happy to hear about it.
* Podman fails to build a 16GB container image (after 30 minutes of downloading dependencies) despite having 90GB free out of a 200GB podman virtual machine
* Podman machine will, for reasons I don't understand, create a filesystem in a block device with wildly different sizes, and it seems like it's just random
* Pushing podman images to a container image registry via the Podman Desktop UI gives no indication that it's doing anything or even recognized the "push image" click, a success or error notification _might_ appear several or tens of minutes later or possibly not at all
* Starting a podman machine might work, but it fails ~75% of the time with not-particularly-exotic options (a bunch of ram and disk) and very cryptic error messages, frequently telling me to file bug tickets (I have)
* Podman Desktop won't let me create a podman machine with more than 44GB of disk, but the podman machine CLI won't let me create a machine with fewer than 100GB (IIRC--it's some number larger than 44, in any case)
Apart from the container image being absurdly large (Python developers love massive packages, I guess), I'm not doing anything exotic.
I always felt it the other way around: docker compose files are weird blobs of YAML that I have to hunt down the location of or parse their under-speced labels to find the location of. I can't make them depend on any non-container services[0], the break my firewall rules[1], and I have to use a whole mess of bespoke tooling just to do normal start/stop/restart operations with them instead of using the same commands I use for literally any other service.
> With quadlets, you delegate everything to systemd. You have to break the configuration up into a bunch of tiny unit files and then separately copy them to /etc or a dedicated user's dotfiles.
The nice thing about quadlets is exactly that, they integrate with systemd and by extension the rest of the system. I don't have to think about `webapp.container` as a "Docker container" I can think of it as just `webapp.service`, like any other piece of software I would install and run. All the related files are in one of the well-speced file locations that follow the same hierarchy as anything else on the system (user -> etc -> /usr), optionally grouped in folders[2].
> Good luck SSH'ing into an unfamiliar system and understanding at a glance what it's doing.
Just use the same tools you'd use on any other systemd system: `systemctl list-units`, `systemctl status`, etc. Versus having to hunt down compose files either manually or by parsing the under-specified labels on the containers.
> (Even moreso if you created dedicated users for each application, which I understand is the recommended solution.)
TBH I've rarely seen this advice. Most people I know just run it as root (which is what I do) or as a `podman` user. But even in this situation it should be pretty easy to figure out whats' running, as you know it's all running as one user and is hard-namespaced to only rely on resources available in that account.
> If I'm just holding it wrong and there exists some better tooling to manage podman in prod that I don't know about, I'm happy to hear about it.
Quadlets are just files that created systemd services, so basically any configuration management or deployment tool will manage them fine. Ansible has a dedicated Quadlet role that works pretty well, or just git clones+`systemctl start`. This would probably be the recommended way if you're not using k8s/etc.
Alternatively, you can just `git clone /etc/containers/systemd/`, `systemctl start container` like with docker compose. If you're running multiple containers, either refer to them with `Wants=`/etc in the Quadlet files, create a `.target` file that references them all, or put them all in a `.pod` and start the pod. I think this is the part were most people stumble though: when you're used to treating containerized software as a separate kind of "thing" it's a little weird to go back to treating it like normal services.
I've been writing something to help with deploying quadlets GitOps-style[3] that will hopefully fill the "more than one server but less than kubernetes" deployment gap.
[0] Unless I wrap the compose steps in a systemd unit, at which point now I have two problems.
[1] Caveat, this has probably gotten better overall but I still run into compose-related firewall issues about once or twice a year
[2] The newer versions of Podman also support `.quadlets` files, that merge all the quadlets into one file.
[3] https://github.com/stryan/materia . There's also https://github.com/orches-team/orches and https://github.com/ubiquitous-factory/quadit
If some BSD would support OCI containers, I would run my apps on BSD.
Yes, you can deploy a Go binary easily with systemd. Could you reliably do this across a fleet of machines? Including managing its configuration, persistent storage, database, network setup, etc.? Maybe, just need Ansible or equivalent config management. What if it were multiple Go binaries? And what if some of them needed to scale up some days because they hit more traffic than the others?
And on and on. Yes, not everyone needs Kubernetes, Nomad or other advanced orchestrators. But comparing them to running a Go binary with systemd is an unfair comparison.
https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
Service file lives in the mono repo where all 6 services live.
Makes it trivial to make changes and redeploy.
We previously built a package manager for Kubernetes to abstract it in the simplest way possible `glasskube install app` but we failed because every abstraction needs to follow a "convention over configuration" pattern at some point. Also, we weren't able to monetize a package manager.
With Distr (https://github.com/distr-sh/distr), we have actually been able to help companies not only package but distribute and either manage or give their customers a way to self-manage applications. Our customers are able to land on-premises contracts at enterprises way faster than before, which is also a clear ROI for paying for Distr.
So, I don't think that you can get the flexibility of a distributed application orchestrator with a simple declarative YAML file if your target environments are diverse.
Just use k8s and follow similar patterns is the conclusion I've arrived at personally.
The inputs (values) are yaml so you can make it look exactly like a Docker Compose file if you want (wouldn't be surprised if there's some charts floating around that do that)
The simplest way to start probably would just be to create a custom Helm chart and define what you want in `values.yaml`.
Is using Docker/Compose "just" as the layer for installing & managing runtime environment and services correct? Especially for languages like PHP?
I.e. am I holding it wrong if I run my "build" processes (npm, composer, etc) on the server at deploy time same as I would without containers? In that sense Docker Composer becomes more like Ansible for me - the tool I use to build the environment, not the entire app.
For the purpose of my question, let's assume I'm building normal CRUD services that can go a little tall or a little wide on servers without caring about hyper scale.
It's perfectly fine, as long as you accept the risks and downsides. Your IP can get ratelimited for Docker Hub. The build process can exhaust resources on the host. Your server probably needs access to internal dev dependencies repository, thus, needs credentials it would not need otherwise. Many small things like that. The advantage is simplicity, and it's often worth the risk.
How? What I'm describing is using Docker less.
> The build process can exhaust resources on the host
Maybe, but I've yet to have a host where that's the case for usual CRUD fare.
> The advantage is simplicity, and it's often worth the risk.
That's basically what I'm evaluating for here.
For bog standard LAMP or similar stack applications, I've not understood the advantage of going through the build-image-then-pull-on-host rigmarole. There's more layers involved there than something like provisioning with Ansible and just having a deploy script to run the usual suspects.
But I have seen that done fairly often, hence was wondering what the point was.
Also adds moving parts to your deploy which increases risk/introduces more failure modes.
Couple things that come to mind
- disk space exhaustion during build
- I/o exhaustion esp with package managers that have lots of small files (npm)
However, on the small/hobby end I don't think it's a huge concern.
> disk, i/o exhaustion
This is why I mentioned specifically for ecosystems like PHP, which are interpreted. I'm specifically asking for that use case.
I'm not building binaries, my "build" steps are actually deployment steps (npm build, composer install, etc) that I'd be running in exactly the same way on the host. The image I'm deploying by definition also contains my source code because I'm not deploying anything compiled.
"Builds" are the same as deploys, so when working with server(s) instead of larger scale deployments, I'm not seeing the benefit of the whole "build image, pull on server" pipeline when I can just ditch the registry and added layers by doing those steps on the server as I would normally in other kinds of scenarios.
But I have seen this in action, which is why I'm wondering if I'm missing something.
The clearer benefit to me seems to be in this scenario to use it as a fast environment provisioning tool.
K8s as small time is overkill for sure but make sure you don't fall into this trap. https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...
What it gets you is a more powerful Docker Compose running on server that you can interact with via kubectl. No SSH, no custom scripts etc. Just kubectl and YAML.
To me, if there's generally fewer than 10 actual active users at any given time and/or you can easily tolerate 30-60m of down time now and then... I'd lean into the simpler option of docker-compose. While I generally think of compose as a dev tool first, it's definitely useful sometimes.
Probably needs a real generalist though which maybe doesn't include your average dev.
Could i survive with 10 seconds of downtime, probably, but I'd really like if I could avoid it.
https://uncloud.run/docs/guides/deployments/rolling-deployme...
They've basically lost the war against Kubernetes but they could easily claim a lot of ground when it's just one more tweak you're adding to your docker-compose file as it scales.
if you use Caddy as your reverse proxy (instead of nginx for example which does not do this), when requests come in and your service is missing because it's being deployed, Caddy waits for a timeout before giving up. this means that visitors during the brief deploy period don't see errors - they just get a slightly longer wait, which often is not obvious depending on how long your service takes to boot.
> docker compose pull && docker compose up -d is a fine command if you are SSH’d into the host. At customer scale—dozens of self-managed environments behind firewalls, each with its own change-control process—that manual process doesn’t scale.
No idea what this 'customer scale' operation is, but it seems like a pretty clear cut candidate for not using docker compose. I also don't think watchtower should be listed there, it's been archived and was never recommended for production usage anyways.
We just use ansible for this part.
Isn't that a Docker thing rather than Docker Compose though? There is a ton more caveats to add if we don't already assume the reader is familiar with the hard edges of Docker, seems the article only focuses on Docker Compose specifically, probably because it'd be very long otherwise :)
> This is the shape Distr lands on
“Lands on”? I like that less.
* Lack of a user-friendly way of managing a Docker Compose installation on a remote host. SSH-forwarding the docker socket is an option, but needs wrappers and discipline.
* Growing beyond one host (and not switching to something like Kubernetes) would normally mean migrating to Swarm, which is its own can of worms.
* Boilerplate needed to expose your services with TLS
Uncloud [1] fixed all those issues for me and is (mostly) Compose-compatible.
If you remember `docker machine`, this is basically the modern version of that.
I've been using portainer for years, it's decent.
This section misses the one thing I was interested in: how do you avoid downtime in a deployment?
I like to write web applications with Perl and Mojolicious, and a deployment is just "hypnotoad app", and then hypnotoad gracefully starts up new worker processes to handle new requests and lets the other ones exit once they've finished handling their in-flight requests.
When I switched to Docker I found that there was no good way to handle this.
edit: thanks to next comment for referencing one
Haven't used it in a while but this thing is also interesting--it supports a bunch of different ways to spin up k8s https://github.com/tilt-dev/ctlptl
Super easy deployment of additional apps, defined completely in one file (incl setup on host, backups, reverse proxy config, etc).
Never found a reason to migrate away. Swarm was already considered dead when I started using it in 2022[1], but the investment was so low and benefits so big, that it was the right choice for me. I think a lot of people are replicating swarm features with compose, losing a lot of time. But hey, to each their own choice!
Using traefik or caddy as proxy.
Docker context for remote access - over Internet or vpn, whatever.
Swarm-cronjob for scheduled things.
Labels for things that need to run in particular places.
So easy.
Personally, k8s is fine, but its an abstraction for building a service architecture, not the thing an end user (developer) should ever use. If you are in a big company and you are using helm or k8s yaml files to roll things out, your infra or platform teams have missed something out.. building the platform!
https://developer.hashicorp.com/nomad
Disclaimer: I used to work for HashiCorp
[1]: https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless...
I ran docker compose in development a lot. Just an easy way to turn on / off 5 different services at once for a project. Over time this was filling up my machine's storage (like 1 TB). Every few months I needed to run docker compose prune and see 600GB free up
https://docs.docker.com/engine/logging/drivers/journald/
I believe Podman can do something similar.
And the error handling was terrible. Most of these problems resulted in a Python stack trace in some docker-compose internals instead of a readable error message. Googling the stack trace usually lead to a description of the actual problem, but that's really not something that inspires confidence.
Things like a good security posture by default, health checks, drift detection, and port forwarding.
Seems like an ad for whatever "Distr" is though; I haven't run into any of these issues with Dokploy and everything's been running fine for months.
Docker/containers are great, especially for local development. But I feel the docker compose model quickly becomes a lot of messy brittle squeeze for little gain when multiple containers need to integrate.
Better then to just take the plunge for the "real deal" and set up a non-HA k8s/k3s cluster with the interactions between the workloads clearly specified.
In other words. I care care more about the interactions declaratively spelled out than the "scale to the moon" HA, auto-scaling, replicas or whatever people get sold on.
And LLMs make this even easier. If you love reviewing yaml manifests....
It's been a great way for us to make sure that developers and CI/CD get exactly the same build environment, mount-points, paths, network access, permissions, etc.
It's been a super solid tool overall, and I'm pretty happy with it. The only thing that would make our setup better would be if we could figure out how to go rootless/daemonless with it.
Very few separate ecosystem transfers are quite that frictionless.
If you want to test something that is between compose and k8s, check ring: https://github.com/kemeter/ring
Docker compose assumes all your services can reach each other over docker, which I find horribly insecure.
I separate all my services by user account at least, maybe even by VM, and I run them all in rootless podman containers. So it just doesn't fit my style, but I'm sure it works fine.
The only issue is the little downtime during deployments.
Ie you need a sysadmin. Oops, you fired them all 10 years ago when the agile devopsing became the best thing after the pumpkin latte.
What I like for my prototype projects is how easy is to use it with podman too.
Granted, its B2B Saas with not many users, maybe 100 concurrent.
80% of workloads dont need the complexity of Kubernetes and run fine with compose.
If you run more than one service/codebase, you might be better suited to using a proper container orchestration platform. Doesn't have to be Kubernetes. AWS ECS, GCP Cloud Run, Kamal are all modern options here.
If you run a single codebase in production, why are you even containerizing? Language ecosystems have done a phenomenal job of improving their dependency management since Docker was released. Python has uv. Go has modules. NodeJS has pnpm. Do you actually get benefit from containerizing if you're deploying to a single production host somewhere?
It's nice to get an easy question every once in a while.
even their follow up - Docker Compose vs Kubernetes.
Docker compose for me has been great - no complexity.