I am excited to share Pathway, a Python data processing framework we built for ETL and RAG pipelines.
https://github.com/pathwaycom/pathway
We started Pathway to solve event processing for IoT and geospatial indexing. Think freight train operations in unmapped depots bringing key merchandise from China to Europe. This was not something we could use Flink or Elastic for.
Then we added more connectors for streaming ETL (Kafka, Postgres CDC…), data indexing (yay vectors!), and LLM wrappers for RAG. Today Pathway provides a data indexing layer for live data updates, stateless and stateful data transformations over streams, and retrieval of structured and unstructured data.
Pathway ships with a Python API and a Rust runtime based on Differential Dataflow to perform incremental computation. All the pipeline is kept in memory and can be easily deployed with Docker and Kubernetes (pipelines-as-code).
We built Pathway to support enterprises like F1 teams and NATO to build mission-critical data pipelines. We do this by putting security and performance first. For example, you can build and deploy self-hosted RAG pipelines with local LLM models and Pathway’s in-memory vector index, so no data ever leaves your infrastructure. Pathway connectors and transformations work with live data by default, so you can avoid expensive reprocessing and rely on fresh data.
You can install Pathway with pip and Docker, and get started with templates and notebooks: https://pathway.com/developers/showcases
We also host demo RAG pipelines implemented 100% in Pathway, feel free to interact with their API endpoints: https://pathway.com/solutions/rag-pipelines#try-it-out
We'd love to hear what you think of Pathway!
