Pathway
Type: full-code · Vendor: Pathway · Language: Python, Rust · License: BSL-1.1 · Status: active · Status in practice: emerging · First released: 2023
Pathway is a Python and Rust framework that builds streaming data and RAG pipelines on a differential-dataflow engine, keeping vector indexes synchronised with their sources as data changes.
Description. Pathway runs the same pipeline code over batch and streaming data, with a Rust engine that recomputes results incrementally as inputs change. Its LLM extension provides parsers, embedders, splitters, and an in-memory real-time vector index, plus document-store components that re-index automatically on new data. Teams use it to build retrieval-augmented generation pipelines whose indexes stay current without a separate batch re-indexing job. The framework integrates with LlamaIndex and LangChain.
Agent loop shape. Pathway is a data-pipeline runtime rather than an agent loop. Connectors ingest documents and change events; the differential-dataflow engine parses, chunks, embeds, and indexes them incrementally; and a vector or document store serves retrieval queries. A question-answering layer retrieves the top documents for a query and passes them to an LLM, with the index kept in sync as sources change.
Primary use cases
- real-time retrieval-augmented generation pipelines
- live document indexing and vector search
- incremental ETL over streaming and batch data
- continuously synchronised knowledge bases for agents
Key concepts
- DocumentStore → streaming-feature-pipeline (docs) — A pipeline component that automatically indexes documents and updates itself when new data arrives, so the served index reflects the source without a separate batch re-indexing job.
- Differential dataflow engine (docs) — A scalable Rust engine that performs incremental computation, recomputing only what changed as inputs update, and runs the same Python pipeline code over both batch and streaming data.
- In-memory real-time Vector Index → vector-memory (docs) — A vector index kept inside the running pipeline that stays synchronised with data sources in real time, serving similarity queries for RAG over live documents.
- AdaptiveRAGQuestionAnswerer → agentic-rag (docs) — A RAG question-answerer that limits the number of documents sent to the LLM chat to save tokens, adjusting retrieval to the query.
Patterns this full-code implements —
- ★Streaming Feature Pipeline
A Python/Rust differential-dataflow framework that processes documents into RAG features incrementally as a continuous stream and keeps a live vector index synchronised with sources in real time, no…
- ★★Agentic RAG
The LLM extension ships question-answering components that retrieve the k best documents and pass them to an LLM, plus an AdaptiveRAGQuestionAnswerer that adjusts retrieval to optimise token usage, o…
- ★★Naive RAG
Pathway's BaseRAGQuestionAnswerer is a standard retrieve-then-generate RAG: given a query it pulls the k best documents from the vector store and sends them with the question to the LLM, with knowled…
Neighbourhood
Click any neighbour to follow the lineage. Scroll to zoom, drag to pan.
Alternatives & relatives
- low-code · visual builder
Streamkap
Streamkap also keeps a vector store in sync with source data in real time, but as a managed CDC streaming service rather than a Python/Rust dataflow framework you write pipelines in.
- full-code · framework
LlamaIndex
Pathway integrates with LlamaIndex; LlamaIndex provides the RAG orchestration and query layer while Pathway supplies the live, incrementally-updated index underneath.
Listed as alternative by (1)
References
pathwaycom/pathway README
“Pathway Live Data Framework provides an LLM extension with all the utilities to integrate LLMs with your data pipelines (LLM wrappers, parsers, embedders, splitters), including an in-memory real-time Vector Index, and integrations with LLamaIndex and LangChain.”
“Pathway Live Data Framework offers real-time data indexes (vector search, full text search, and more) allowing you to effortlessly synchronize your index with data sources in real time.”
“it is easy to create in Pathway a pipeline serving as a `DocumentStore`, which automatically indexes documents and gets updated upon new data.”
Provenance
- Last analyzed:
- Last updated:
- Verification status: partial