Self-RAG: Teaching LLMs to Decide When to Retrieve

Updated
9 min read
Self-RAG: Teaching LLMs to Decide When to Retrieve
TL;DR

Self-RAG introduces reflection tokens that let the model decide whether retrieval is needed and evaluate passage relevance and citation support, outperforming standard RAG on factuality benchmarks.

The Problem With Always Retrieving

Standard RAG retrieves documents for every query - even when the model already knows the answer with high confidence. This wastes compute, introduces latency, and can actually hurt performance when retrieved passages are irrelevant or distracting. A question like "What is 2+2?" does not benefit from retrieval. Self-RAG (arXiv:2310.11511) teaches the model to decide for itself.

Reflection Tokens

Self-RAG adds four special token types to the model's vocabulary:

  • RETRIEVE: Should retrieval happen for this query? [Yes / No]
  • ISREL: Is this retrieved passage relevant to the query? [Relevant / Irrelevant]
  • ISSUP: Does the generated text cite this passage accurately? [Fully supported / Partially supported / No support]
  • ISUSE: Is this response overall useful? [5-point scale]

These tokens are generated by the model itself as part of the output, enabling dynamic decision-making without any external classifier.

Four-Stage Training

Self-RAG training has four stages:

  1. Critic training: Train a separate critic model on human-annotated data for each reflection token type (using GPT-4 annotations to scale).

  2. Data augmentation: Use the critic to annotate a large SFT corpus - for each training example, determine whether retrieval would help, annotate retrieved passages with ISREL/ISSUP/ISUSE tokens.

  3. SFT on augmented data: Fine-tune the base language model on the augmented corpus with all reflection tokens interleaved.

  4. Inference: At test time, the model generates RETRIEVE tokens to invoke retrieval, evaluates passages, generates conditioned on selected passages, and outputs self-evaluation tokens alongside the response.

python
# Pseudocode for Self-RAG inference
def self_rag_generate(model, retriever, query):
    # First, predict whether to retrieve
    retrieve_token = model.predict_retrieve(query)

    if retrieve_token == "Yes":
        passages = retriever.retrieve(query, k=5)
        # Score passage relevance
        relevant_passages = [
            p for p in passages
            if model.predict_isrel(query, p) == "Relevant"
        ]
        # Generate conditioned on relevant passages
        response = model.generate(query, context=relevant_passages)
        # Self-evaluate support and usefulness
        support = model.predict_issup(query, response, relevant_passages)
        score = model.predict_isuse(query, response)
    else:
        # Generate without retrieval
        response = model.generate(query)

    return response

Inference-Time Control

The ISUSE scores enable beam search over multiple candidate responses. At inference time, you can set thresholds: only output responses with ISUSE >= 4, or prefer responses with full citation support. This lets you trade off diversity against factuality without retraining.

Benchmark Results

Self-RAG outperforms ChatGPT on factuality benchmarks (TriviaQA, PopQA, ARC-Challenge, MedQA) without retrieval for simple questions - the model correctly identifies when retrieval is unnecessary. On complex knowledge-intensive tasks, it outperforms standard RAG by selectively retrieving only when beneficial. It generates citations for claims it makes, enabling verification.

Further Reading

#self-rag#rag#retrieval#reflection-tokens#adaptive

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.