RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

The standard RLHF pipeline has three stages. First, a language model is supervised fine-tuned (SFT) on high-quality demonstrations. Second, human annotators compare pairs of model responses and a reward model (RM) is trained on their preferences. Third, the SFT policy is optimized with reinforcement learning using the RM as a proxy for human judgment, with a KL divergence penalty to prevent the policy from drifting too far from the original model.

The second stage is the bottleneck. Collecting human preferences is expensive — roughly $0.67 per example for the summarization task. Annotators must be trained, they disagree with each other (inter-annotator agreement is only 73–77%), and the process takes weeks. For every new task or domain, you start over. As models get better, they need more nuanced feedback, which requires more skilled — and more expensive — annotators.

The core idea is simple: instead of asking a human "Which response is better?", ask a large language model. Given a context (e.g. a Reddit post) and two candidate responses (e.g. two summaries), the LLM is prompted to evaluate and express a preference. The prompt has four parts: a preamble describing the task, optional few-shot exemplars, the sample to annotate, and an ending that elicits the preference.

Rather than extracting a binary label, the system computes a soft preference distribution by taking the softmax of the log-probabilities of generating the tokens "1" and "2". For example, a preference of [0.6, 0.4] means the LLM slightly prefers the first response. This soft label conveys richer information than a hard one-hot choice.

Two critical techniques improve the quality of AI-generated labels:

Position bias mitigation. LLMs can be biased toward whichever response appears first (or second). To counter this, each pair is scored twice with the order reversed, and the results are averaged. Smaller models show dramatically more position bias — PaLM 2 XS prefers the same position 56% of the time even after swapping, compared to only 18% for PaLM 2 L.

Chain-of-thought reasoning. Before emitting a preference, the LLM is asked to explain its reasoning. This two-step approach — first generate a rationale, then score — consistently improves alignment with human preferences by up to +1.9%.

Once the AI-generated preferences are collected, the standard RLHF pipeline resumes. A reward model is trained on the soft AI labels using cross-entropy loss — the same loss used in RLHF, except the targets are soft distributions instead of hard labels.

The RL phase uses REINFORCE with a baseline value function, initialized from the SFT checkpoint. The optimization objective balances reward maximization against KL divergence from the SFT policy:

Canonical RLAIF still trains a separate reward model — a process that takes time and introduces a "staleness" problem. As the RL policy improves, its outputs drift away from the data the RM was trained on, causing the RM's scores to become unreliable. Iteratively retraining the RM is one fix, but it is expensive.

Direct-RLAIF (d-RLAIF) offers an elegant alternative: skip the reward model entirely and use the off-the-shelf LLM as a live scorer during RL. For each generated response, the LLM is prompted to rate its quality on a scale of 1–10. The probability of each score token is computed, a weighted average gives the raw score, and finally the score is normalized to [−1, 1] and used directly as the reward signal.

This eliminates both the staleness issue and the time cost of RM training. In experiments, d-RLAIF matched or outperformed canonical RLAIF on every task. On summarization, d-RLAIF achieved a 74% win rate over SFT, compared to 68% for same-size canonical RLAIF.

The paper evaluates on three tasks — summarization (Reddit TL;DR), helpful dialogue generation, and harmless dialogue generation — using human evaluation as the gold standard.

For summarization, RLAIF achieves a 71% win rate over SFT while RLHF achieves 73% — a difference that is not statistically significant. In a direct head-to-head comparison, RLAIF and RLHF are preferred at equal rates by human evaluators (50% win rate).

For helpful dialogue, the pattern holds: RLAIF at 63% and RLHF at 64% over SFT, again not statistically different.

For harmless dialogue, RLAIF actually outperforms RLHF, achieving a harmless rate of 88% compared to RLHF's 76% and SFT's 64%.

A key question is how the quality of the AI labeler affects downstream performance. The paper finds a clear scaling relationship: larger LLM labelers produce better-aligned preferences. PaLM 2 L achieves 78% alignment with human labels, PaLM 2 S drops to 73.8%, and PaLM 2 XS falls to 62.7%.

But here is the surprising finding: even when the AI labeler is the same size as the policy being trained (PaLM 2 XS labeling for PaLM 2 XS policy), RLAIF still achieves a 68% win rate over SFT — only slightly below the 71% achieved with a much larger labeler. This is a step toward self-improvement: a model improving itself using its own judgments.

The d-RLAIF setup pushes this even further. For helpful dialogue, the LLM providing rewards and the starting policy are the exact same model checkpoint — and it still improves. This constitutes a strict example of LLM self-improvement.

The paper systematically studies three prompt variations for AI labeling:

Chain-of-thought (CoT) consistently improves alignment. By asking the LLM to first explain its reasoning before expressing a preference, alignment improves by +1.3% to +1.9% depending on the task. This mirrors findings from other reasoning tasks — thinking before answering helps.

Detailed preambles help for complex tasks (summarization gains +1.3%) but show mixed results for simpler tasks like judging helpfulness or harmlessness, where the task is already intuitive for the LLM.

Few-shot examples surprisingly hurt performance on summarization and helpfulness, with alignment monotonically decreasing as exemplar count increases. Only the harmlessness task benefits from in-context learning (+2.7% with 2-shot). The hypothesis is that the LLM already understands these tasks well enough that exemplars become distracting rather than helpful.

Beyond matching quality, RLAIF dramatically reduces cost. The paper estimates that AI labeling costs approximately $0.06 per example (using GPT-4 pricing at the time), compared to $0.67 per example for human annotation through Google Cloud's labeling service. That is over 10x cheaper — and the AI labeler works around the clock without fatigue, training time, or inter-annotator calibration.

Since the AI labeler in canonical RLAIF is only used once to generate the preference dataset (not called during RL), using an even larger, more capable AI labeler is not prohibitively expensive. The cost is amortized across all the preferences it labels.

Manual inspection of summarization outputs revealed interesting differences between RLAIF and RLHF policies. In some cases, RLHF hallucinated plausible-sounding details not present in the source text — for example, stating a user was "20 years old" when no age was mentioned. RLAIF summaries were less prone to this.

On the other hand, RLAIF sometimes produced less fluent summaries — run-on sentences or repeated phrases like "How do I get over this?" that did not reflect the original text. However, when human annotators blindly ranked the policies on accuracy, coverage, and coherence, the differences were not statistically significant.

RLAIF's deepest contribution is not a specific method — it is a proof of concept. It showed that the boundary between human judgment and AI judgment, for the purpose of training reward models, is far thinner than assumed. This opened the door to a new generation of alignment methods that reduce or eliminate human annotation entirely, from DPO to self-play to Constitutional AI's iterative refinement. The practical implication is clear: alignment at scale is no longer gated by the speed of human annotators.