Nginx introduces native support for ACME protocol (opens in new tab)

A practical problem with DNS-01 is that every DNS provider has a different API for creating the required TXT record. Certbot has more than a dozen plugins for different providers, and the list is growing. It shouldn't be nginx's job to keep track of all these third-party APIs.

It would also be unreasonable to tell everyone to move their domains to a handful of giants like AWS and Cloudflare who already control so much of the internet, just so they could get certificates with DNS-01. I like my DNS a bit more decentralized than that.

no need to wait: https://en.angie.software/angie/docs/configuration/modules/h...

(angie is the nginx fork lead by original nginx developers that left f5)

But you have to have your dns api key loaded and many dns providers don’t allow api keys per zone. I do like it but a compromise could be awful.

The problem with DNS-01 is that you can only use one delegation a time. I mean, if you configure a wildcard cert with _acme-challenge.example.com in Google, you couldn't use it in Cloudflare, because it uses a single DNS authorization label (subdomain).

The solution has been evolving along these years and currently the las IETF draft is https://datatracker.ietf.org/doc/draft-ietf-acme-dns-account...

The new proposal brings the dns-account-01 challenge, incorporating the ACME account URL into the DNS validation record name.

One of Traefik's shortcomings with ACME is that you can only use one api key per DNS provider. This is problematic if you want to restrict api keys to a domain, or use domains belonging to two different accounts. I hope Nginx will not have the same constraint.

I use dns01 in my homelab with step-ca with caddy. It's a joy to use

Yes, ACME-DNS please - https://github.com/joohoi/acme-dns

Lego supports it.

I don't even know why anyone wouldn't use the DNS challenge unless they had no other option. I've found it to be annoying and brittle, maybe less so now with native web server support. And you can't get wildcards.

Switch to Angie then. It supports DNS-01 very well.

How does NGINX fit into that though?

Does DNS-01 support DNS-over-HTTPS to the registered domain name servers? If so, then it should be extremely simple to extend nginx to support DNS claims; if not, perhaps DNS-01 needs improvements.

Why would nginx ever need support for the DNS-01 challenge type? It always has access to `.well-known` because nginx is running an HTTP server for the entire lifecycle of the process, so you'd never need to use a lower level way of doing DV. And that seems to violate the principle of least privilege, since you now need a sensitive API token on the server.

What I really like about Caddy is their better syntax. I actually use nginx (via nginx proxy manager) and Traefik but recently I did one project with Caddy and found it very nice. I might get the time to change my selfhosted setup to use Caddy in the future but probably will go with something like pangolin [1] because it provides alternative to cloudflare tunnels too.

[1] https://github.com/fosrl/pangolin

I switched over to caddy recently. Nginx' non-information about the http 1 desync problem drove me over. I'm not going to wait for something stupid to happen or an auditor ask me questions nginx doesn't answer.

Caddy is really easier than nginx. For starters, I now have templates that cover the main services and their test services, and the special service that runs for an education institution. Logging is better. Certificate handling is perfect (for my case, at least). And it has better metrics.

Now I have to figure out plugins though, because caddy doesn't have rate limiting and some stupid bug in powerbi makes a single user hit certain images 300.000 times per day. That's a bit of a downside.

Definitely. I use traefik for some stuff at home and will likely swap it out now.

It's also been in Apache since 2018

I felt the same but switched to Caddy for my reverse proxy last year and have had a great experience.

Admittedly this was on the back of trying to use nginx-unit, which was an overall bad experience, but ¯\_(ツ)_/¯

Hey! Great to see you here.

I tried dokku (and still am!) and it is so hard getting started.

For reference, - I've used Coolify successfully where it required me to create a Github app to deploy my apps on pushes to master - I've written GH actions to build and deploy containers to big cloud

This page is what I get if I want to achieve the same, and it's completely a reference book approach - I feel like I'm reading an encyclopedia. https://dokku.com/docs/deployment/methods/git/#initializing-...

Contrast it with this, which is INSTANTLY useful and helps me deploy apps hot off the page: https://coolify.io/docs/knowledge-base/git/github/integratio...

What I would love to see for Dokku is tutorials for popular OSS apps and set-objective/get-it-done style getting started articles. I'd LOVE an article that takes me from baremetal to a reverse proxy+a few popular apps. Because the value isn't in using Dokku, it's in using Dokku to get to that state.

I'm trying to use dokku for my homeserver.

Ideally I want a painless, quick way to go from "hey here's a repo I like" to "deployed on my machine" with Dokku. And then once that works, peek under the hood.

I must say this is something that showcases NixOS very well.

This is all it takes to start a nginx server. Add this block and everything starts up perfectly first time, using proper systemd sandboxing, with a certificate provisioned, and with a systemd timer for autorenewing the cert. Delete the block, and it's like the server never existed, all of that gets torn down cleanly.

  services.nginx = {
    enable = true;
    virtualHosts = {
      "mydomain.com" = {
        enableACME = true;
        locations."/" = {
          extraConfig = ''; # Config goes here
        };
      };
    };
  }

I recently wanted to create a shortcut domain for our wedding website, redirecting to the SaaS wedding provider. The above made that a literal 1 minute job.

> I would really appreciate if someone has documented this extensively for docker compose

Run `certbot certonly` on the host once to get the initial certs, and choose the option to run a temporary server rather than using nginx. Then in `compose.yml` have a mapping from the host's certificates to the nginx container. That way, you don't have to touch your nginx config when setting up a new server.

You can then use a certbot container to do the renewals.

E.g.

  nginx:
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt

  certbot:
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"

In your nginx.conf you have

    ssl_certificate /etc/letsencrypt/live/$DOMAIN/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/$DOMAIN/privkey.pem;

And also

    location /.well-known/ {
        alias /usr/share/nginx/html/.well-known/;
    }

For the renewals.

This is mostly why I run nginx outside of Docker, I've written about it here: https://nickjanetakis.com/blog/why-i-prefer-running-nginx-on...

I keep these things separate on the servers I configure:

    - Setting up PKI related things like DH Params and certs (no Docker)
    - My app (Docker)
    - Reverse proxy / TLS / etc. with nginx (no Docker)

This allows configuring a server in a way where all nginx configuration works over HTTPS and the PKI bits will either use a self-signed certificate or certbot with DNS validation depending on what you're doing. It gets around all forms of chicken / egg problems and reduces a lot of complexity.

Switching between self-signed, Let's Encrypt or 3rd party certs is a matter of updating 1 symlink since nginx is configured to read the destination. This makes things easy to test and adds a level of disaster recovery / reliability that helps me sleep at night.

This combo has been running strong since all of these tools were available. Before Let's Encrypt was available I did the same thing, except I used 3rd party certs.

What's the issue with nginx-proxy? We've used it for years to handle CI deploying multiple multiple Docker compose Apps to the same server [1] without issue, with a more detailed writeup at [2].

This served us well for many years before migrating to use Kamal [3] for its improved remote management features.

[1] https://docs.servicestack.net/ssh-docker-compose-deploment

[2] https://servicestack.net/posts/kubernetes_not_required

[3] https://docs.servicestack.net/kamal-deploy

I personally just terminate TLS at nginx, run nginx directly on the metal, and all the services are containerized behind it. I suspect if I had nginx then proxying to remote nodes I'd probably just use an internal PKI for that.

Usually the solution is to either not add ssl until you have the certs, or use selfsigned/snakeoil placeholder certs to get nginx started.

Personally I use dns everywhere. I have a central server running dehydrated and dns challenges every night which then rsyncs to all the servers (I'm going to replace it with vault). I kind of like having one place to check for certs

> you need certificates to start the NGINX server but you need the NGINX server to issue certificates?

I just pre-populate with a self-signed cert to start, though I'd have to check how to do that in docker.

The same config also works with certbot. I've used it for years.

This is really cool, but I find projects that have thousands of people depending on it not cutting a stable release really distasteful.

Edit: Downvote me all you want, that's reality folks, if you don't release v1.0.0, the interface you consume can change without you realizing it.

Don't consume major version 0 software, it'll bite you one day. Convince your maintainers to release stable cuts if they've been sitting on major version 0 for years. It's just lazy and immature practice abusing semantic versioning. Maintainers can learn and grow. It's normal.

Dehydrated has been major version 0 for 7 years, it's probably past due.

See also React, LÖVE, and others that made 0.n.x jumps to n.x.x. (https://0ver.org)

CalVer: "If both you and someone you don't know use your project seriously, then use a serious version."

SemVer: "If your software is being used in production, it should probably already be 1.0.0."

https://0ver.org/about.html

Caddy did it almost a decade ago. IIRC it had some form of automatic Let’s Encrypt HTTPS back in 2016.

So Nginx is just about 9 to 10 years late. Lol

And the brilliant thing about open source projects is that if someone felt it was so important to have it built-in, they could have done so many years ago.

Given that Caddy has a history that includes choices like "refuse to start if LE cannot be contacted while a valid certificate exists on disk" I'm pretty happy to keep my certificate issuance separate from a web server.

I need a tool to issue certs for a bunch of other services anyway, I don't really see how it became such a thing for people to want it embedded in their web server.

I'm currently working on getting the Trixie packages uploaded. It'll be there this week.

As you've said Debian 13 was released 4 days ago - it takes some time to spin up the infrastructure for a new OS (and we've been busy with other tasks, like getting nginx-acme and 1.29.1 out).

(I work for F5)

That's Debian's fault I guess

You could always switch to the Angie fork if you require the DNS challenge type with the wildcard domains:

https://en.angie.software/angie/docs/configuration/modules/h...

> allowing internally-signed certificates to be valid via said Intermediary

By design, nothing is allowed to delegate signing authority, because it would become an immediate compromise of everything that got delegated when your delegated authority got compromised. Since only CAs can issue certs, and CAs have to pass at least some basic security scrutiny, clients have assurance that the thing giving it a cert got said cert from a trustworthy authority. If you want a non-trustworthy authority... go with a custom CA. It's intentionally difficult to do so.

> If your load balancer, proxy server, web server, or router appliance can’t mint me a basic Acme certificate via DNS-01 challenges, then you officially suck and I will throw your product out for something like Caddy the first chance I get.

I mean, that's a valid ask. It will become more commonplace once some popular corporate offering includes it, and then all the competitors will adopt it so they don't leave money on the table. To get the first one to adopt it, be a whale of a customer and yell loudly that you want it, then wait 18 months.

What company that has enough infrastructure to dictate an IT Department is also only using certificates on their web servers, and thus doesn't have a standard tool for issuing/renewing/deploying certificates for *all* services that need them?

DNS challenge is complicated by the fact that every registrar has their own API. HTTP is easier for nginx because it’s a single flow and it already does HTTP.

I’m sure nginx will get DNS but it’s still an open question when it will support your particular registrar or if at all.

It added ACME in 3.2, the DNS challenge is coming next: https://www.haproxy.com/blog/announcing-haproxy-3-2#acme-pro...

This optional module makes simple cases simpler.

Having distinct tools for serving content and handling certs is not a problem, and nothing changes on this side. Moreover, the module won't cover every need.

BTW, cerbot is rather a "fat tool" compared to other acme tools like lego. I've had bad experiences with certbot in the past because it tried to do too much automatically and it's hard to diagnose – though I think certbot has been rewritten since then, since it has no more dependency on python zope.

It's kind of annoying to set up. Last I remember certbot could try to automatically configure things for you but unless you had the most default setup it wouldn't work. Just having Nginx do everything for you seems like a better solution.

I wonder about the same thing. I've come to the conclusion that it's driven a lot by Management-Ideal definition of devops: developers who end up doing OPs without sufficient knowledge or experience to do it well.

Nginx with certbot is annoying to setup. Especially with HTTP challenge. Mostly because of a circular dependency. You need nginx to clear the challenge and once verboten gets a cert you need to reload nginx.

I switched to Lego because it has out of the box support for my domain registrar so I could use DNS instead of HTTP challenge. It’s also a single go binary which is much simpler to install than certbot.

Nginx itself is mostly just a collection of modules, and it's up to the one building/packaging the nginx distribution to decide what goes in it. By default, nginx doesn't even build the ssl or gzip modules (though thankfully it does build the http module by default). Historically it only had static modules, which needed to be enabled or disabled at compile time, but now it has dynamic modules that can be compiled separately and loaded at runtime. Some older static modules now have the option of being built as dynamic modules, and new modules that can be written as dynamic modules generally are. A distro can choose to package a new dynamic module in their base nginx package, as a separate package, or not at all.

In a typical distro, you would normally expect one or more virtual packages representing a profile (minimal, standard, full, etc) that depends on a package providing an nginx binary with every reasonable static-only module enabled, plus a number of separately packaged dynamic modules.

Yes, that is correct.

Yes, a few lines in the config and you don’t need certbot any more.

You can specify any ACME API base URL. It’s not just Let’s Encrypt.

We have been using Caddy for many years now. Picked just because it has automatic cert provisioning. Caddy is really an easier alternative, secure out of the box.

Nginx maintains their own repository from which you can install nginx on your Ubuntu / Debian systems.

I looked at Arch and they're a version behind, which surprised me. Must not be a heavily maintained arch package.

I assume they're complaining that it's a snap vs flatpak, not so much vs the distro package repos.

You don’t need exactly the same cert for failover. You only need a valid certificate. You don’t even need the same cert for every entry in your load balancer. Client will pick a single IP address when resolved, then connect to it and will keep using that TLS connection for the whole session.

Maybe it's better these days, but even as an experienced systems administrator, I found certbot _incredibly_ annoying to use in practice. They tried to make it easy and general-purpose for beginners to web hosting, but they did it with a lot of magic that does Weird Stuff to your host and server configuration. It probably works great if you're in an environment where you just install things via tarball, edit your config files with Nano, and then rarely ever touch the whole setup again.

But if you're someone who needs tight control over the host configuration (managed via Ansible, etc) because you need to comply with security standards, or have the whole setup reproducible for disaster recovery, etc, then solutions like acme.sh or LEGO are far smaller, just as easy to configure, and in general will not surprise you.

Certbot is a giant swiss army chainsaw that can do everything middlingly well, if you don't mind vibecoding your encryption intrastructure. But a clean solution it usually isn't.

(That said, I'm not too thrilled by this implementation. How are renewals and revocations handled, and how can the processes be debugged? I hope the docs get updated soon.)

From the seeming consensus I was dreading setting let's encrypt up on nginx, until I did it and it was and has been... Completely straightforward and painless.

Maybe if you step off the happy path it gets hairy, but I found the default certbot flow to be easy.

From a quick look it seems like a command you use to reconfigure nginx? And that's separate from auto-renewing the cert, right?

Maybe not hard, but Caddy seems like even less to think about.

Certbot is a utility that can only be installed via snap. That crap won’t make it to our servers, and many other people view it the same way I do.

So this change is most welcome.

absolute nightmare to get this to work inside docker compose dude. Nobody has documented a decent working solution for this yet. Too many quirks and third parties like nginx-proxy-manager or nginx-proxy/nginx-proxy on github make it even more terrible

You'd get rate limited pretty hard by Let's Encrypt, but if you're rolling your own acme servers you could do it this way.

If you wanted to use LE though, you could use a more "traditional" cert renewal process somewhere out-of-band, and then provision the resulting keys/certs through whatever coordination thing you contrive (and HUP the nginxs)

They don't need to share a single cert. Multiple certificates can be, and possibly should, issued for the same address (or set of addresses). This means that one front door server that gets popped doesn't expose all connections to the larger service.

Downside is obviously certificate maintenance increases, but ACME automated the vast majority of that work away.

Yes. If you need it earlier you will have to use a separate ACME client.

Here's the docs for the Angie's version of the http_acme module:

https://en.angie.software/angie/docs/configuration/modules/h...

The original announcement of Angie ACME:

Angie, fork of Nginx, supports ACME - https://news.ycombinator.com/item?id=39838228 - March 27, 2024 (1 comment)

Per above, it looks like ACME support was released with Angie 1.5.0 on 2024-03-27.

BTW, if you don't care about ACME, and want the original nginx, then there's also the freenginx fork, too:

Freenginx: Core Nginx developer announces fork - https://news.ycombinator.com/item?id=39373327 - (1131 points) - Feb 14, 2024 (475 comments)

I have no use for this either, but I’m sure a lot of people will enjoy not having to install a separate ACME client.

I grew up on Apache and eventually became a wizard with its configuration and myriad options and failures modes. Later on, I got semi-comfortable with nginx which was a little simpler because it did less than Apache but you could still get a fairly complex configuration going if you're running weird legacy PHP apps for example.

When I tried using Caddy with something serious for the first time, I thought I was missing something. I thought, these docs must be incomplete, there has to be more to it, how does it know to do X based on Y, this is never going to work...

But it DID work. There IS almost nothing to it. You set literally the bare minimum of configuration you could possibly need, and Caddy figures out the rest and uses sane defaults. The docs are VERY good, there is a nice community around it.

If I had any complaint at all, it would be that the plugin system is slightly goofy.

Caddy has been great for me. I don't think you should switch if your current setup works but give it a try in a new project.

I like it!!! I am using Apache mod_md on Debian for personal project. That is working fine but when setting up a new site it somehow required two Apache restarts which is not super smooth