Better Airflow with Metaflow (opens in new tab)

(outerbounds.com)

8 pointsvtuulos3y ago3 comments

3 comments

I have used Airflow as a product for over a year and I can say for sure that Airflow sucks for orchestrating ML.

The ability to pass state in Airflow absolutely sucks. It's interesting that Metaflow is trying to compile into Airflow but I am skeptical about it's scalability on Airflow especially with DAGs that have a larger task footprint.

savin-goyal3y ago

Metaflow passes state using an object store (s3, azure blob store, etc.) even within Airflow - short circuiting Airflow's xcom machinery. But agreed, Airflow presents many more scalability challenges - this integration addresses a few in-place as well as preserves the ability to swap out Airflow with a more scalable workflow orchestrator if needed.

vtuulosOP3y ago

Totally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow

j / k navigate · click thread line to collapse

3 comments

chse_cake3y ago

I have used Airflow as a product for over a year and I can say for sure that Airflow sucks for orchestrating ML.

savin-goyal3y ago

vtuulosOP3y ago

Totally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow

j / k navigate · click thread line to collapse