i dont really understand concept of SPoF. My windows servers have never really failed me. In case machine reboots in the middle of your batch - Task scheduler can restart your job. In case the job fails, it can retry after a certain interval.

you can configure your job to retry unless it returns success - and unless someone nukes your windows machine - it will get executed. if you are afraid that your windows machine will get nuked - then you can use SQL Server Agent on High Availability Cluster - and it will do the job.

if you want your jobs as a code - you can code them in Powershell and store in git/azure devops repo. You can deploy your jobs with the same powershell, or even use a CI/CD pipeline to do that.

Someone else who has the same problem!

Looks like airflow, celery and every other workflow orchestrator doesn't want to deal with it and just asks you to deal with it using shit like k8s.

I decided to write a simple scheduler that runs in a separate thread in the background of our webapp, so that it can be parceled into our existing elastic beanstalk deployment.

I use two database tables to pick up tasks and run them, and have some amount of failover. Just need to be a bit careful around deadlocks, but thats a cakewalk compared to the dumpster fire that is configuring these Babylon tower frameworks.

If you're interested I can write up some sample scheduler code and publish it which shouldn't be more than a few hundred lines and do what we're looking for.