I am Jan, CTO and one of the creators of Pathway, the real-time data processing framework. I’m excited to share Pathway’s ready-to-use AI Pipelines, configurable with just YAML! These frameworks offer out-of-the-box solutions for AI search, RAG, and more—optimized for real-time indexing and in-memory processing.
What makes it simple? YAML templates! The pipeline templates are fully customizable using YAMLs to fit your needs, from changing the data sources to the choice of the LLM model, all without touching Pathway’s Python code. Thanks to the Pathway data processing engine, these pipelines are optimized for real-time, updating the pipeline by ingesting the data as it arrives.
Why it matters: * YAML templates: that’s it, no Python code required. * Full customization: Configure everything, from LLM models and indexes to tokenizers and retrieval strategies. * Real-time data sync: Connect to data sources like Google Drive, SharePoint, and more, with pipelines automatically updating to reflect all additions, deletions, and changes in real-time. * Powerful search capabilities: Internal in-memory indexing supports vector, hybrid, and full-text search. The indexing is automatically maintained as data sources update in real-time. * Deploy in minutes: Quickly deploy on-premises or on cloud platforms like AWS, GCP, or Azure. * It's open: don't hesitate to contribute!
Templates: https://pathway.com/developers/templates GH repo: https://github.com/pathwaycom/llm-app/ Docs: https://pathway.com/developers/ai-pipelines/welcome
Deploy your personalized AI pipeline in minutes and let us know what you think!
