Haystack 2.0: The Production RAG Pipeline Framework by deepset

What Is Haystack?

Haystack is an open-source NLP framework by deepset designed specifically for building production retrieval-augmented generation (RAG) pipelines. Version 2.0, released in 2024, was a complete redesign that made the framework more modular and production-ready.

While LangChain and LlamaIndex are broad frameworks covering many LLM use cases, Haystack is narrower and deeper: it's built around the retrieval pipeline as the core abstraction, and it shows in the quality of the retrieval-specific tooling.

Component-Based Architecture

Every Haystack pipeline is a directed acyclic graph of components. Each component has typed inputs and outputs, making pipelines composable and testable in isolation.

The four key component types:

• DocumentStore - the vector database (Elasticsearch, OpenSearch, Weaviate, Qdrant, Chroma, in-memory)
• Retriever - fetches relevant documents (BM25, dense, hybrid)
• Ranker - re-ranks retrieved documents for relevance
• Generator - the LLM that produces the final answer

from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import RAGPromptBuilder
from haystack.document_stores.in_memory import InMemoryDocumentStore

document_store = InMemoryDocumentStore()
# ... index documents ...

pipeline = Pipeline()
pipeline.add_component("retriever", InMemoryBM25Retriever(document_store=document_store))
pipeline.add_component("prompt_builder", RAGPromptBuilder(template=prompt_template))
pipeline.add_component("llm", OpenAIGenerator(model="gpt-4o-mini"))

pipeline.connect("retriever", "prompt_builder.documents")
pipeline.connect("prompt_builder", "llm")

result = pipeline.run({"retriever": {"query": "What is RAG?"}})

YAML Pipeline Definition

Pipelines can be serialized to YAML for version control and deployment:

components:
  retriever:
    type: haystack.components.retrievers.in_memory.InMemoryBM25Retriever
    init_parameters:
      document_store: !component InMemoryDocumentStore
  llm:
    type: haystack.components.generators.OpenAIGenerator
    init_parameters:
      model: gpt-4o-mini
connections:
  - sender: retriever.documents
    receiver: prompt_builder.documents

Hybrid Retrieval

Haystack's hybrid retrieval combines BM25 (keyword) and dense (embedding) retrieval, then merges results with a reciprocal rank fusion algorithm. This consistently outperforms either method alone, especially for technical documents where both exact term matching and semantic understanding matter.

Evaluation Framework

Haystack includes a built-in evaluation framework with metrics for retrieval quality (MRR, MAP, NDCG) and generation quality (faithfulness, answer relevance). This lets you objectively measure whether a pipeline change actually improved performance.

deepset Cloud

For teams that want managed pipelines, deepset Cloud wraps Haystack in a hosted service with monitoring, A/B testing, and deployment management.

Haystack vs LangChain vs LlamaIndex

Haystack is more opinionated and production-focused. LangChain has a broader ecosystem and more integrations. LlamaIndex has deeper support for complex index structures. If your primary use case is a retrieval-heavy production pipeline, Haystack's evaluation framework and YAML-defined pipelines are a significant operational advantage.

Resources

• Haystack GitHub
• Official docs
• Tutorials