I'm currently building pgflow, which is a simple, postgres-first engine that uses task queues to perform real work.
Have explicit DAG approach, strong typesafety, nice DSL in TypeScript and a dedicated task queue worker that allows it to run solely on Supabase without any external tools.
I'm super close to the alpha release, if you guys want more info, check out the readme for SQL core (https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#rea...) or my Twitter (https://x.com/pgflow_dev).
Hope that grabs someone attention :-) Cheers
I'm very close to releasing an alpha, will post here when ready!
That said, my impression is that Airflow is a really dated choice for a greenfield project. There isn't a clear successor though. I looked into this recently, and was quickly overwhelmed by Prefect, Dagster, Temporal, and even newer ones like Hatchet and Hamilton
Most of these frameworks now have docs / plugins / sister libraries geared around AI agents
It would be really helpful to read a good technical blog doing a landscape of design patterns in these different approaches, and thoughts on how to fit things together well into a pipeline given various quirks of LLMs (e.g. nondeterminism).
This page is a good start, even if it is written as an airflow-specific how-to!
Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe 2gb out of 8gb system RAM in use. No reason is given in the logs.
I am trying to switch to dagster, not because it's awesome, but because it hasn't crashed randomly on me.
Their operational perspective is catastrophic; how does one view the logs for a dag through the UI[1]? Why can't it store the python in the database they have attached to their deployment, versus making me jump through 80,000 hoops to put the files in the right magic directory on disk of every worker[2]?
1: no, not <https://airflow.apache.org/docs/apache-airflow/stable/ui.htm...> I mean the log, you know, like in the old days of $(tail -f /var/log/the.thing). I'm open to the answer hiding somewhere in this gobbledygook <https://airflow.apache.org/docs/apache-airflow/stable/admini...> but who is the target audience for having such a fancy UI and omitting log viewing from it, doubly so if there's some alleged http just for viewing logs
2: https://airflow.apache.org/docs/apache-airflow/stable/core-c... and double-plus-good anytime python software mentions PYTHONPATH -- that's how you know you're in for a hot good time https://airflow.apache.org/docs/apache-airflow/stable/admini...
https://www.dbos.dev/blog/durable-execution-crashproof-ai-ag...
You either get simplicity with the caveate that your systems need to perfectly align.
Or you get complexity but will work with basically anything (airflow).
Disclosure, I'm the CEO of DBOS.
[0] https://github.com/dbos-inc/dbos-transact-py
[1] https://www.dbos.dev/blog/durable-execution-coding-compariso...
If you want a product hint from me, I think that adding integrations natively into the platform that would allow vibe coders to build asynchronous agents easier would really boost revenue. Like email, text, etc.
Probably not your vision, just a suggestion
[1]: Limited to what decorator arguments can do. I suspect it could become an issue with `@task.branch` if some post-processing would be needed to adjust for smaller models' finickinesses.
[2]: As the final step is described at the top of the function.
It is _potentially_ more restrictive than writing pure Python functions, but the plus side is that we can interject certain Airflow-specific features into how the agent runs. And this isn't mean for someone who knows agents inside & out / wants the low-level customizability.
The best example of this today is log groups: Airflow lets you log things out as part of a "group" which has some UI abstractions to make it easier. This SDK takes the raw agent tool calls and turns them each into a log group, so you can see a) at a high level what the agent is doing, and b) drill down into a specific tool call to understand what's happening within the tool call.
To your point about the `@task.llm_branch`, the SDK & Pydantic AI (which the SDK uses under the hood) will re-prompt the LLM up to a certain number of attempts if it receives output that isn't the name of a downstream task. So there shouldn't be much finickiness.
It just exposes set of functions to propagate the DAG through states and queues tasks for a task worker to perform the actual work and acknowledge completion or failure back to the sql orchestrator.
I was working on it for last few months and it will be ready in upcoming weeks, first version is dedicated to work on Supabase but I plan to make it agnostic.
If you want to learn more, check out the SQL Core readme which explains the whole concept (https://github.com/pgflow-dev/pgflow/tree/main/pkgs/core#rea...) or my Twitter for updates and some demos (https://x.com/pgflow_dev).
In real life program, I don't think that you will have hundreds of calls to LLM or agent in your app so much that you have any code gains to decorator but at the opposite the decorator will make it very hard to have parametric values or values not hard coded but from config that you don't set up upfront at application startup like globals. That is a bad practice...
Airflow actually uses decorators to indicate something is an explicit task in a data pipeline vs just a utility function, so this follows that pattern!
It also uses an "operator" under the hood (Airflow's term for a pre-built, parameterized task) which can be subclassed and customized if you want to do any customization.
