I tried setting up nomad but couldn't figure out service discovery, external ingress / letsencrypt in the time I allotted myself...
Caddy for reverse proxy and tls handling running as a service, so all ingress goes through that, there wasn't too much exciting there. I was porting over a docker-compose workload, so most services have a static port and I just route to the port in the caddyfile (I passed my router as the DNS for the reverse proxy, so its by hostname). I'll get caddy to use nomad for service addresses at some point.
But for photoprism, I've got the service discovery set up to its database, it's kind of awkward since the nomad native discovery goes into env vars or a file through the `template` block [2], but it does work.
Nomad is pickier about setting resource limits, so I had to actually set those to something reasonable.
I've mostly got Docker based services, so this reference [3] has been useful. There's a couple ways to mount the volumes, which is annoying, and there's some gotchas around docker image handling (short shorty is don't use the `latest` tag).
I probably haven't organized my jobs/groups/tasks well (the analog to k8s pods hierarchy), but that's for later.
[0] https://developer.hashicorp.com/nomad/tutorials/get-started/... [1] https://developer.hashicorp.com/nomad/tutorials/enterprise/p... [2] https://www.hashicorp.com/blog/nomad-service-discovery [3] https://developer.hashicorp.com/nomad/docs/drivers/docker
Just so I understand - this means deploying a new service involves building a docker image, deploy it, then manually update the caddyfile (ie manual ingress)?
> There's a couple ways to mount the volumes, which is annoying,
Fwiw this is annoying on multi-node docker swarm too - i even consider proper volume support to be one of the strongest arguments for considering k8s even for somewhat simple setups.
And I'd agree that the volume story isn't great compared to what I know of k8s. There's info out there for setting up NFS volumes, or something like portworx or ceph, but that's going beyond what I want to do for 1-3 pet nodes on a home server, I can deal with volumes staying bound to a node.
[0] https://traefik.io/blog/traefik-proxy-fully-integrates-with-...