I recently took over an Airflow system from a former colleague, and in our case, it’s just overly complex for what’s really a pretty simple data flow.
* Luigi -- extensive usage (4y+)
* Makefiles -- (15y+)
* GitHub Actions -- (4y+)
* Airflow -- little usage (<6 months)
* Dagster -- very little, just trying it out
* Prefect -- just followed tutorial
Although it lacks a lot of the monitoring and advanced web ui other platforms have (maybe because of it), Luigi is the simplest to reason about IMHO.
For a new project that will require complex orchestrations, I'd probably go with Dagster or Prefect nowadays. Dagster seems more complex and more powerful with its data lineage functionality, but I have very little experience with either tool.
If it's a simple project, a mix of Makefiles + GH Actions can work well.
Dagster and even Luigi feel like overkill but I'd still like to plug those into a unified interface where I can view previous runs, mainly logs and exit codes. Being able to do some light job configuration or add retries would be nice but not required. For the moment I just use a logging handler that writes to a database table and that's fine
Honestly, just use airflow, it has its issues but it sucks in well known and predictable ways.
The tools you describe all have the endpoint "you can't get there from here" and the only difference is if it takes you 5 seconds, 5 minutes, 5 days, 5 weeks or 5 months to learn that.
[1] https://dvc.org
What I loved was its simplicity + its out of the box features. To set it up just took a simple MS SQL DB + An Installer. Bam you are up and running an absolute rock solid scheduler(i've seen million+ jobs running on it without it breaking a sweat). Then you could install (or use it to deploy) execution agents to all the servers you wanted as workers.
It also installed a robust Desktop GUI that had so many services built in ready to go (anything from executing scripts all the way to performing direct actions against countless products a company would have or against various cloud services).
There were so many pre built actions where all you had to do was input credentials and it would enumerate the appropriate properties from that service automatically. Then you could connect things together (ie, pull something from the cloud, process it on some other server, store it, pass it along to another service, whatever you wanted)
Only problem was this is very much a B2B application and their sales is really only interested in selling to enterprises and not end users. I really wish we had something like this that regular people could download.
Everything ive seen listed here requires extensive setup,requires coding, or does not have a robust desktop GUI but instead some half baked web gui which might require dropping back down to scripts/coding. You could set up hundreds/thousands of automated steps in ActiveBatch without writing a single line of code. I miss that product.
Mind you, it might have been just culture at that place, but I don't think this is as good of an example as you make it be. Sure, it was easy to get started and made the life easier at the beginning, but running it at scale was not in any way easy.
Seriously, took me much less time than setting up airflow. Even had a webpage in the end, with all the tasks, a tree view, downstream, upstream tasks (these were incremental improvements beyond the initial half-day), CLI... The works.
I now know the points of fragility I didn't know before, but I'd do it again.
However, it’s simple only if you are already familiar with software containers and Kubernetes. But it’s perhaps better to learn than having to deal with dependency hell in Python or Java.
It's a bit expensive but the only thing they push you to do is to upgrade managed K8S and Airflow once for a while.
