Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

Retrieval-Augmented Generation (RAG) was a major step forward: instead of relying entirely on memorized knowledge, the model consults external documents. But standard RAG has a rigid pipeline — retrieve $K$ documents, prepend them, generate. Three problems arise:

First, retrieval happens indiscriminately. Whether the question needs factual grounding or is a creative writing prompt, the model always retrieves. This can inject irrelevant or distracting context.

Second, the model is never trained to check whether retrieved passages actually support its output. It may generate text that contradicts the evidence, producing a fluent but factually wrong answer with false citations.

Third, retrieval is one-shot: the model retrieves once at the beginning and never revisits the decision. If the first retrieval misses, there is no second chance.

Self-RAG rethinks the entire RAG pipeline. Instead of a fixed retrieve-then-generate pipeline, Self-RAG trains a single language model to perform three actions as an integrated loop, segment by segment:

Step 1 — Decide: Given the input and any preceding text, the model predicts a special Retrieve token. If retrieval is needed, it outputs Retrieve=Yes and calls a retriever. If not, it outputs Retrieve=No and generates freely from its own knowledge.

Step 2 — Generate with evidence: When retrieval is triggered, the retriever returns $K$ passages. The model processes all $K$ in parallel, generating a candidate continuation from each. For each passage, it first predicts an IsRelevant token — is this passage actually useful for the query?

Step 3 — Critique: After generating each candidate segment, the model predicts two more reflection tokens: IsSupported (is the output actually backed by the passage?) and IsUseful (is this a helpful response overall?). A segment-level beam search then selects the best continuation based on weighted scores from all three critique dimensions.

The key insight is that all of these decisions — retrieve, assess relevance, verify support, rate utility — are made by the same language model as part of its normal next-token prediction, using special tokens added to its vocabulary.

The central innovation of Self-RAG is the reflection token system. These are special tokens added to the model's vocabulary — not natural language feedback, but discrete control signals that the model learns to generate as part of its output. There are four types:

Retrieve decides whether the next segment needs retrieval. Its values are Yes, No, or Continue (keep using the current passage). Think of it as the model raising its hand to say "I need to look something up."

IsRelevant evaluates whether a retrieved passage provides useful information for the input. Values: Relevant or Irrelevant. This acts as a filter — if the passage is irrelevant, the model knows not to trust it.

IsSupported checks whether the generated output is actually backed by the passage. Values: Fully Supported, Partially Supported, or No Support / Contradictory. This is the factuality guard — it catches the model when it invents claims that go beyond the evidence.

IsUseful rates the overall quality of the response on a 1–5 scale, independently of whether it is factual. A response could be fully supported but still unhelpful if it misses the point of the question.

Self-RAG's training has two stages. The first stage trains a critic model that can predict reflection tokens. The second stage uses the critic to annotate a large training corpus, then trains the generator model on this annotated data.

Stage 1 — Training the critic: Manually labeling reflection tokens for every segment would be prohibitively expensive. Instead, the authors prompt GPT-4 with type-specific instructions (e.g., "Given this instruction and output, decide if retrieval would help") and collect 4k–20k labeled examples per token type. These labels are then used to fine-tune a smaller in-house critic model (Llama 2 7B) via standard language modeling. The resulting critic achieves over 90% agreement with GPT-4 predictions — meaning the authors distill GPT-4's judgment into a small, efficient model they fully control.

Stage 2 — Training the generator: Once the critic is trained, it is used offline to annotate the full training corpus. For each input-output pair, the critic decides whether retrieval is needed, evaluates retrieved passages for relevance, checks whether the output is supported, and rates overall utility. All of these reflection tokens are inserted directly into the text. The generator model is then trained on this enriched corpus using the standard next-token prediction objective:

At inference time, Self-RAG generates text segment by segment (typically sentence by sentence). At each segment step, the process unfolds as follows:

If Retrieve=Yes, the retriever returns $K$ passages. The model generates $K$ candidate segments — one from each passage — in parallel. Each candidate is scored by a weighted sum of the critique token probabilities:

This design makes Self-RAG controllable without retraining. A practitioner deploying the model for a medical QA system can increase the IsSupported weight to demand strong evidence grounding. The same model deployed for creative writing can reduce retrieval frequency by raising the threshold $\delta$ for triggering Retrieve=Yes.

The retrieval threshold itself is adaptive. Instead of always retrieving when Retrieve=Yes has the highest probability, Self-RAG checks whether the normalized probability of Retrieve=Yes exceeds a threshold $\delta$. A higher $\delta$ means less retrieval — useful for tasks where parametric knowledge suffices.

Self-RAG was evaluated on six diverse tasks: two closed-set tasks (PubHealth fact verification and ARC-Challenge reasoning), two open-domain QA tasks (PopQA and TriviaQA), and two long-form generation tasks (biography generation evaluated with FactScore, and ASQA evaluated for correctness, fluency, and citation accuracy).

Self-RAG 7B and 13B consistently outperform baselines across all tasks. On PopQA, Self-RAG 13B achieves 55.8% accuracy, beating retrieval-augmented ChatGPT (50.8%) and Llama2-chat 13B (51.8%). On PubHealth, Self-RAG 13B reaches 74.5%, far surpassing ChatGPT (70.1%). Most strikingly, on biography generation, Self-RAG 7B achieves a FactScore of 81.2, outperforming the 65B-parameter CoVE model (71.2).

On ASQA citation metrics, Self-RAG 7B achieves 66.9% citation precision and 67.8% recall — approaching ChatGPT's citation recall (76.6%) while exceeding its precision (65.1%). This shows Self-RAG generates claims that are genuinely backed by cited evidence, not just plausible-sounding citations.

The ablation studies reveal which components matter most. Removing the critic entirely (No Critic) — training on data always augmented with the top passage but without reflection tokens — causes major drops, especially on ASQA (from 32.1 to 18.1 str-em). This shows that blindly prepending retrieved text is insufficient; the model needs to learn when retrieval is useful and when it is not.

Removing the retriever (No Retriever) and training without any retrieved passages also hurts, confirming that retrieval does provide value. But the gap from removing the critic is larger — suggesting that the self-reflection mechanism matters more than retrieval alone.

At inference time, using only the top-1 passage (like standard RAG) instead of processing multiple candidates in parallel causes notable drops on PopQA and ASQA. And removing the IsSupported score from beam search specifically hurts citation quality, confirming that each critique dimension provides independent value.

Self-RAG introduced three ideas that shifted how the community thinks about retrieval-augmented generation:

First, adaptive retrieval: the model decides when retrieval is helpful rather than always retrieving. This preserves the versatility of the base LM — creative tasks skip retrieval entirely, factual tasks retrieve aggressively.

Second, integrated self-critique: instead of relying on an external verifier or reward model at inference time, the model's own generation contains quality assessments. This makes verification a built-in capability rather than an add-on.

Third, inference-time controllability: by adjusting the weights on critique token scores, practitioners can tune the same model for different applications without retraining. High citation precision for legal or medical applications, high fluency for creative writing — all from the same checkpoint.

These ideas point toward a future where language models are not just generators but active agents that monitor their own reliability, consult external knowledge only when needed, and produce outputs that carry their own evidence trail.

Self-RAG sits at a turning point in the retrieval-augmented generation landscape. Before it, retrieval was a fixed preprocessing step. After it, retrieval became a learnable, adaptive decision embedded within the generation process itself. The reflection token framework showed that self-assessment does not require expensive reinforcement learning — it can be learned through supervised distillation and exercised through standard next-token prediction.