I used NGINX for many years, but around 1.5 year ago I switched to Caddyserver as the SSL certificates are just so nice getting them autohandled. I noticed a slight performance decrease, but for my kinda services it is not so important.
But I am curious, what do you use? and more importantly why?
This separates concerns completely:
- HAProxy knows about and manages the TLS, balancing, client routing etc;
- Varnish knows about and manages response caching and ESI processing (and often a combination of both);
- Apache knows about and runs the various backend services (a php web app, a couple of ruby third party tools, etc)
Nginx has some significant downsides to what we currently use, unless we opt for the paid version which best I can tell is ~$1K/instance/month. These aren't hypothetical differences these are features we actually use:
- no sync for load balancing data (sticky peer data, rate limit data, etc): HAProxy supports this out of the box;
- no active health checks: HAProxy supports this out of the box;
- no API for purging cache: Varnish supports this out of the box.
- no ESI support: Varnish supports this out of the box. Best I can tell even the paid version of nginx doesn't support this.
And another question? Do you use varnish on edge? In case, how do you do this? I normally just use bunnycdn for caching.
Our "edge" is CF right now, but that's likely to change (coincidentally to bunny) because our traffic model doesn't really fit their offering very well.
This actually goes to answer your first question: we've progressively moved more things into our own origin stack because what CF offers us doesn't work out (and in some cases probably wouldn't be easily solved with nginx either).
As an example: CF and Nginx both have basic rate limiting, but AFAIK, neither can track two separate rates out of the box, if at all. We use a relatively high req/period limit, and then a much lower threshold error/period limit which counts any requests that result in a client-caused error (i.e. 4xx errors).
I'm pretty certain we would lose flexibility if we "simplified" by either offloading some aspects to a service, or even if we tried to consolidate several parts into a kitchen-sink tool like nginx.
We mostly use edge (i.e. CDN) caching the same way we expect a browser cache to work (but across users): stuff that we absolutely know isn't changing (i.e. we reference asset (img, js, css) content using URLs based on a content hash). We use our internal Varnish cache for application content, combined with a job in our queue that can purge that content (selectively) when required. Could we use a commercial CDN for that part? Sure in theory, but it's just another thing we won't have control over, and have to develop specific to the CDN we're using at the time, subject to the rules of their system - and it still wouldn't necessarily to ESI.
Caddy for local development. Less config and setup.
- Autoprovison SSL
- Serve static files without another web server
- Rewrites, redirects, map HMR, websocket ports etc
Sure nginx works too but Caddy is definitely less work and less hassle
- as a layer on top the app servers for not having to expose Node.js, and loadbalancing app servers,
- brotli_static,
- serving avif conditionally[1]
- anonymizing IPs in logs
- injecting the caching headers
- injecting the CSP header
- SSL Offloading
I mean, the autorenew bots need more priviledges, such as:
- HTTP challenges need to be via HTTP (not HTTPS) [2],
- HTTP challenges need write permissions on a servable directory,
- DNS or HTTP challenges would need a program on a live server,
- need ‘pass out’ firewall exceptions without IP scope. "We don’t publish a list of IP addresses we use to validate… Let’s Encrypt" [3]
2. https://datatracker.ietf.org/doc/html/rfc8555#section-8.3
3. https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let...
Here’s a blog post https://blog.uxtly.com/isolated-tls-certificate-creation