Haystack 2.0: The Production RAG Pipeline Framework by deepset

Updated
8 min read
Haystack 2.0: The Production RAG Pipeline Framework by deepset
TL;DR

Haystack 2.0 from deepset offers a component-based pipeline architecture for building production RAG systems, with YAML-defined pipelines and a built-in evaluation framework.

What Is Haystack?

Haystack is an open-source NLP framework by deepset designed specifically for building production retrieval-augmented generation (RAG) pipelines. Version 2.0, released in 2024, was a complete redesign that made the framework more modular and production-ready.

While LangChain and LlamaIndex are broad frameworks covering many LLM use cases, Haystack is narrower and deeper: it's built around the retrieval pipeline as the core abstraction, and it shows in the quality of the retrieval-specific tooling.

Component-Based Architecture

Every Haystack pipeline is a directed acyclic graph of components. Each component has typed inputs and outputs, making pipelines composable and testable in isolation.

The four key component types:

  • DocumentStore - the vector database (Elasticsearch, OpenSearch, Weaviate, Qdrant, Chroma, in-memory)
  • Retriever - fetches relevant documents (BM25, dense, hybrid)
  • Ranker - re-ranks retrieved documents for relevance
  • Generator - the LLM that produces the final answer
python
from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import RAGPromptBuilder
from haystack.document_stores.in_memory import InMemoryDocumentStore

document_store = InMemoryDocumentStore()
# ... index documents ...

pipeline = Pipeline()
pipeline.add_component("retriever", InMemoryBM25Retriever(document_store=document_store))
pipeline.add_component("prompt_builder", RAGPromptBuilder(template=prompt_template))
pipeline.add_component("llm", OpenAIGenerator(model="gpt-4o-mini"))

pipeline.connect("retriever", "prompt_builder.documents")
pipeline.connect("prompt_builder", "llm")

result = pipeline.run({"retriever": {"query": "What is RAG?"}})

YAML Pipeline Definition

Pipelines can be serialized to YAML for version control and deployment:

yaml
components:
  retriever:
    type: haystack.components.retrievers.in_memory.InMemoryBM25Retriever
    init_parameters:
      document_store: !component InMemoryDocumentStore
  llm:
    type: haystack.components.generators.OpenAIGenerator
    init_parameters:
      model: gpt-4o-mini
connections:
  - sender: retriever.documents
    receiver: prompt_builder.documents

Hybrid Retrieval

Haystack's hybrid retrieval combines BM25 (keyword) and dense (embedding) retrieval, then merges results with a reciprocal rank fusion algorithm. This consistently outperforms either method alone, especially for technical documents where both exact term matching and semantic understanding matter.

Evaluation Framework

Haystack includes a built-in evaluation framework with metrics for retrieval quality (MRR, MAP, NDCG) and generation quality (faithfulness, answer relevance). This lets you objectively measure whether a pipeline change actually improved performance.

deepset Cloud

For teams that want managed pipelines, deepset Cloud wraps Haystack in a hosted service with monitoring, A/B testing, and deployment management.

Haystack vs LangChain vs LlamaIndex

Haystack is more opinionated and production-focused. LangChain has a broader ecosystem and more integrations. LlamaIndex has deeper support for complex index structures. If your primary use case is a retrieval-heavy production pipeline, Haystack's evaluation framework and YAML-defined pipelines are a significant operational advantage.

Resources

#haystack#rag#pipeline#deepset#production

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.