LlamaIndex is an open-source data framework engineered to connect custom data sources with large language models through production-ready Retrieval-Augmented Generation (RAG) pipelines. This LlamaIndex RAG guide focuses specifically on data ingestion, semantic chunking, document indexing, vector store persistence, and contextual query synthesis, rather than general-purpose agent orchestration. The guide covers how to architect end-to-end RAG workflows in Python, implement semantic splitting, optimize multi-document routing, and evaluate retrieval precision.
LlamaIndex RAG Guide: Architectural Differences vs. LangChain
Developers evaluating tools for LLM applications frequently compare LlamaIndex and LangChain. While both libraries interact with embedding models and vector databases, their core design philosophies serve fundamentally different stages of application architecture.
| Evaluation Metric | LlamaIndex | LangChain |
|---|---|---|
| Primary Focus | Data ingestion, indexing, and advanced retrieval | Generalized agent logic and tool orchestration |
| Core Abstraction | Index, Node Parser, Query Engine, Retriever | Chain, Agent, Runnable, Tool |
| Document Loaders | SimpleDirectoryReader and 100+ LlamaHub connectors | Document loaders across community packages |
| Chunking Intelligence | Native Sentence, Token, Markdown, and Semantic splitters | Character and RecursiveTextSplitter |
| Response Synthesis | Refine, Compact, Tree Summarize, and Accumulate | Map-reduce, Refine, and Stuff documents chains |
| Query Orchestration | Native SubQuestionQueryEngine and RouterEngine | AgentExecutor and LangGraph state machines |
| Ideal Use Case | Knowledge bases, document QA, and enterprise search | Autonomous multi-step agents and complex logic flows |
LangChain is built to orchestrate actions and chains of execution. In contrast, LlamaIndex treats retrieval as the central challenge. In production architectures, top engineering teams often combine both - using LlamaIndex as the specialized retrieval engine and LangChain or CrewAI as the higher-level reasoning supervisor.
Environment Setup and Package Installation
Modern LlamaIndex uses a modular package structure that isolates core abstractions from provider-specific dependencies. Install the core framework alongside the official OpenAI integrations:
pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai llama-index-readers-file
If you plan to utilize local Hugging Face sentence transformers or production vector databases such as Qdrant vector database, Chroma, or Pinecone, install the corresponding integration modules:
pip install llama-index-vector-stores-qdrant llama-index-embeddings-huggingface llama-index-retrievers-bm25
Data Ingestion with SimpleDirectoryReader
Loading unstructured documents across varying file formats typically requires tedious parsing scripts. LlamaIndex resolves this through SimpleDirectoryReader, as outlined in the official documentation for LlamaIndex documents and nodes, which inspects files by extension and parses text, PDFs, Markdown, HTML, and Word files automatically.
import os
from llama_index.core import SimpleDirectoryReader
# Ingest documents recursively with metadata extraction
reader = SimpleDirectoryReader(
input_dir="./enterprise_docs",
recursive=True,
required_exts=[".pdf", ".md", ".txt"],
file_metadata=lambda filename: {
"source_file": os.path.basename(filename),
"department": "Engineering" if "eng" in filename else "Operations"
}
)
documents = reader.load_data()
print(f"Loaded {len(documents)} document pages successfully.")
Each loaded document encapsulates raw textual content alongside structured metadata keys. Preserving metadata at ingestion time is crucial for downstream filtering and source attribution.
Node Parsing and Semantic Chunking Strategies
A common mistake in RAG engineering is relying on naive character-based text splitting. Arbitrary character cuts sever sentences midway, splitting crucial context across chunk boundaries and degrading embedding quality.
LlamaIndex decomposes documents into atomic entities called Nodes. By applying context-aware parsers, you preserve semantic meaning and document relationships.
from llama_index.core.node_parser import SentenceSplitter, SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding
# Strategy 1 - Sentence Splitter with Chunk Overlap
sentence_parser = SentenceSplitter(
chunk_size=512,
chunk_overlap=64
)
sentence_nodes = sentence_parser.get_nodes_from_documents(documents)
# Strategy 2 - Semantic Chunking based on embedding similarity drops
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
semantic_parser = SemanticSplitterNodeParser(
buffer_size=1,
breakpoint_percentile_threshold=95,
embed_model=embed_model
)
semantic_nodes = semantic_parser.get_nodes_from_documents(documents)
The semantic chunker calculates cosine similarity between consecutive sentences, placing chunk boundaries only where topical transitions occur. This significantly boosts retrieval precision for legal agreements, technical manuals, and multi-topic articles.
Hierarchical Node Parsing and Parent Document Retrieval
When chunking text, small chunks (128 to 256 tokens) yield higher embedding precision during cosine similarity search, but lack surrounding context when generating an answer. Conversely, large chunks (1024 tokens) dilute embedding focus.
Hierarchical node parsing resolves this dilemma by storing small child nodes for similarity search while linking back to larger parent nodes for LLM synthesis.
from llama_index.core.node_parser import HierarchicalNodeParser, get_leaf_nodes
from llama_index.core.storage.docstore import SimpleDocumentStore
# Create multi-tier chunk hierarchy
hierarchical_parser = HierarchicalNodeParser.from_defaults(
chunk_sizes=[2048, 512, 128]
)
nodes = hierarchical_parser.get_nodes_from_documents(documents)
leaf_nodes = get_leaf_nodes(nodes)
# Store all nodes in document store
docstore = SimpleDocumentStore()
docstore.add_documents(nodes)
During retrieval, an AutoMergingRetriever looks at the retrieved leaf nodes. If more than 50 percent of child nodes belonging to a parent node are retrieved, it automatically swaps the child nodes for the complete parent node, feeding coherent paragraphs to the LLM.
Building and Persisting VectorStoreIndex
Once documents are transformed into nodes, they must be converted into OpenAI text embeddings and indexed for approximate nearest neighbor retrieval using the VectorStoreIndex architecture.
Global Configuration with Settings
Use the global Settings object to standardize embedding models, LLMs, and chunking parameters throughout your application:
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
Settings.llm = OpenAI(model="gpt-4o", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512
Settings.chunk_overlap = 64
In-Memory vs Production Vector Database Storage
For local testing, default in-memory indices suffice. For production environments handling millions of document chunks, integrate a dedicated vector store such as Qdrant:
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient
# Initialize Qdrant Client
client = QdrantClient(host="localhost", port=6333)
vector_store = QdrantVectorStore(client=client, collection_name="knowledge_base")
# Bind vector store to storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)
# Construct index and persist embeddings
index = VectorStoreIndex(
semantic_nodes,
storage_context=storage_context
)
print("Index constructed and stored in Qdrant.")
Query Engines and Response Synthesis Modes
Retrieval is only half the battle. Once relevant nodes are fetched, the response synthesizer structures how retrieved text is passed to the LLM to generate the final response.
from llama_index.core import get_response_synthesizer
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
# Configure retriever to fetch top 5 relevant nodes
retriever = VectorIndexRetriever(
index=index,
similarity_top_k=5
)
# Select synthesis mode
synthesizer = get_response_synthesizer(response_mode="compact")
query_engine = RetrieverQueryEngine(
retriever=retriever,
response_synthesizer=synthesizer
)
response = query_engine.query("What are the service level agreements for enterprise uptime?")
print(f"Response: {response.response}\n")
# Inspect source attribution nodes
for node in response.source_nodes:
print(f"Node ID: {node.id_} | Score: {node.score:.4f} | Source: {node.metadata.get('source_file')}")
Common Response Synthesizer Modes
- compact - Fits as many retrieved text chunks as possible into the prompt window before issuing an LLM call. This reduces token overhead and minimizes latency.
- refine - Queries the LLM with the first context chunk, then sequentially passes each subsequent chunk to update and refine the draft answer.
- tree_summarize - Builds a bottom-up hierarchical summary tree from retrieved nodes, ideal for queries requiring macro-level document summaries.
- accumulate - Executes independent queries against each context node and returns an aggregated list of answers.
Hybrid Search with BM25 Keyword and Dense Vector Retrieval
Dense vector embeddings capture conceptual similarity, but they struggle with exact keyword lookups like part numbers, error codes, and legal identifiers. Combining sparse BM25 retrieval with dense vector embeddings delivers the best precision.
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import QueryFusionRetriever
# 1. Initialize sparse keyword retriever
bm25_retriever = BM25Retriever.from_defaults(
nodes=semantic_nodes,
similarity_top_k=5
)
# 2. Initialize dense vector retriever
dense_retriever = VectorIndexRetriever(
index=index,
similarity_top_k=5
)
# 3. Fuse retrievers using Reciprocal Rank Fusion (RRF)
hybrid_retriever = QueryFusionRetriever(
retrievers=[bm25_retriever, dense_retriever],
similarity_top_k=5,
num_queries=1,
mode="reciprocal_rerank",
use_async=True
)
hybrid_query_engine = RetrieverQueryEngine.from_args(
retriever=hybrid_retriever,
response_mode="compact"
)
hybrid_response = hybrid_query_engine.query("What does error code ERR_4091 indicate?")
print(hybrid_response)
By blending sparse frequency matching with high-dimensional vector representations, your pipeline accurately resolves specific technical nomenclature without forfeiting broad semantic understanding.
Advanced Query Engines for Sub-Questions and Multi-Source Routing
Real-world enterprise inquiries rarely map neatly to a single search query. Users frequently ask comparative questions such as: "How does our authentication latency in Q3 compare to Q4?". LlamaIndex provides advanced query engines to decompose and route complex queries automatically.
Decomposing Queries with SubQuestionQueryEngine
from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata
# Wrap individual indices as tools
q3_tool = QueryEngineTool(
query_engine=q3_index.as_query_engine(),
metadata=ToolMetadata(
name="q3_metrics",
description="Contains infrastructure performance metrics for Q3"
)
)
q4_tool = QueryEngineTool(
query_engine=q4_index.as_query_engine(),
metadata=ToolMetadata(
name="q4_metrics",
description="Contains infrastructure performance metrics for Q4"
)
)
# Construct SubQuestionQueryEngine
sub_engine = SubQuestionQueryEngine.from_defaults(
query_engine_tools=[q3_tool, q4_tool],
use_async=True
)
comparison_response = sub_engine.query(
"Compare the authentication response time and error rate between Q3 and Q4."
)
print(comparison_response)
The query engine decomposes the prompt into sub-questions, executes parallel vector lookups across both tools, and synthesizes the unified answer with verified source citations.
Evaluation with RAGAS and TruLens
A RAG pipeline without rigorous automated evaluation degrades rapidly under production workloads. Unchecked hallucination, context drift, and irrelevant node retrieval alienate users.
The Core RAG Evaluation Metrics
Production RAG assessment relies on three core evaluation dimensions established in RAGAS evaluation metrics and TruLens open source evaluation:
- Context Relevance - Evaluates whether retrieved nodes contain information relevant to the user prompt without extraneous noise.
- Groundedness (Faithfulness) - Verifies that every assertion made by the generator is provably derived from the retrieved context chunks.
- Answer Relevance - Measures whether the generated answer directly addresses the intent of the question.
Implementing Evaluation with RAGAS
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevance, context_precision
from datasets import Dataset
# Format evaluation samples from LlamaIndex query outputs
data_samples = {
"question": ["What is the refund policy duration?"],
"answer": [str(response)],
"contexts": [[node.get_content() for node in response.source_nodes]],
"ground_truth": ["Refunds must be requested within 30 days of initial purchase."]
}
dataset = Dataset.from_dict(data_samples)
eval_results = evaluate(
dataset=dataset,
metrics=[faithfulness, answer_relevance, context_precision]
)
print(f"Faithfulness Score: {eval_results['faithfulness']:.2f}")
print(f"Context Precision Score: {eval_results['context_precision']:.2f}")
Integrating evaluation passes into continuous integration pipelines prevents regressions when modifying chunk sizes, prompt templates, or embedding models.
Production Optimization Checklist
To ensure high performance and predictable costs, follow these proven architectural guidelines:
- Leverage Metadata Filtering - Add structured tenant IDs, project namespaces, and timestamps during document ingestion to narrow the vector search space before running similarity math.
- Implement Re-Ranking - Pair initial high-recall vector search (top 25 nodes) with a cross-encoder re-ranker (such as Cohere or BGE) to refine top 5 nodes before LLM prompt injection.
- Stream Query Responses - Use
streaming=Trueon the query engine to emit tokens incrementally, slashing perceived latency for end users. - Cache Embedding Vectors - Cache node embeddings to avoid redundant API billing during incremental document re-indexing.
- Monitor Query Drift - Track low similarity scores and unanswered queries to discover knowledge gaps in your document corpus.
Explore Related Engineering Guides
Continue mastering modern AI infrastructure and full-stack development with our specialized technical playbooks:
- Advanced RAG Architecture and Retrieval Techniques - Master hybrid search, HyDE, and hierarchical retrieval.
- RAG Evaluation with RAGAS and TruLens - Implement production evaluation metrics and hallucination detection.
- Vector Databases for Enterprise RAG - Scale high-dimensional similarity search with Qdrant.
- Explore Software Development Services - Partner with Pristren to design, build, and deploy production AI applications.
Accelerate Your AI Engineering with Pristren
Designing and scaling enterprise-grade RAG pipelines requires deep expertise in data processing, vector infrastructure, and LLM optimization. Pristren partners with product teams to build resilient knowledge systems, autonomous agent workflows, and scalable digital solutions. Explore how our engineering team can master your AI architecture and deliver production-ready intelligence. Schedule an engineering consultation or discover our software engineering services.




Comments
Have a question or something to add? Sign in to join the discussion.