Also, it makes distro packaging dreadful, since you can either ship nothing and be useless for nearly everyone, or ship everything and surprise users if they switch to the official builds and find out stuff is missing.
Nginx used to be as bad, but that's been fixed recently.
Personally Caddy is also not very useful for me, since I use a reverse proxy, and Caddy didn't seem very helpful there the last time I tried. Oh, and it would have been nice to be able to make it generate self-signed certificates for staging environments.