Direct Preference Optimization: Your Language Model Is Secretly a Reward Model

Before DPO, aligning a language model with human preferences followed a three-stage recipe that the field called RLHF:

Stage 1 — Supervised Fine-Tuning (SFT). Start from a pre-trained language model and fine-tune it on high-quality demonstration data. This produces a policy $\pi_{\text{SFT}}$ that can follow instructions but doesn't yet reflect nuanced human preferences.

Stage 2 — Reward Modeling. Collect preference data: for each prompt $x$, two candidate responses are sampled and a human labels which one is better ($y_w \succ y_l$). A reward model $r_\phi(x, y)$ is trained to predict these preferences using the Bradley-Terry model — the probability that response $y_1$ is preferred over $y_2$ is $\sigma(r(x, y_1) - r(x, y_2))$.

Stage 3 — RL Optimization. Use PPO to maximize the learned reward while staying close to the SFT policy via a KL divergence penalty. This requires holding four models in memory: the policy being trained, the frozen reference, the reward model, and a value network for variance reduction.

The RLHF objective asks: find the policy $\pi$ that maximizes expected reward while staying close to the reference policy $\pi_{\text{ref}}$:

$\max_\pi \; \mathbb{E}_{x \sim \mathcal{D}, \, y \sim \pi}[r(x, y)] - \beta \, D_{\text{KL}}[\pi(y|x) \| \pi_{\text{ref}}(y|x)]$

This looks like it needs RL to solve. But Rafailov et al. noticed something remarkable: this optimization has a closed-form solution. The optimal policy is:

$\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\!\left(\frac{1}{\beta} r(x,y)\right)$

where $Z(x)$ is a normalizing constant. Now comes the crucial trick: rearrange this equation to solve for the reward:

$r(x, y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$

The reward is fully determined by the ratio of the optimal policy to the reference policy. The partition function $Z(x)$ cancels when we compare two responses to the same prompt — and that's exactly what the Bradley-Terry model does.

Substituting the implicit reward into the Bradley-Terry preference model gives us the DPO loss — a binary cross-entropy objective that directly optimizes the policy:

Read it from the inside out. The term $\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ is the log-ratio of the policy to the reference — how much the model has drifted from the starting point for a given response. The difference of two such log-ratios is the implicit reward margin: how much more the model rewards the preferred response versus the rejected one. The sigmoid wraps this margin into a probability, and the log turns it into a cross-entropy loss.

The gradient has an elegant interpretation. It increases the probability of $y_w$ and decreases the probability of $y_l$, but weighted by how wrong the model currently is. If the model already strongly prefers $y_w$, the gradient is small — no need to push further. If it incorrectly prefers $y_l$, the gradient is large. This automatic curriculum means DPO focuses its updates where they matter most.

The entire derivation rests on the Bradley-Terry model of preferences, a classic from 1952. It says: the probability that item A is preferred over item B depends on the difference in their "strengths" (here, rewards):

$P(y_1 \succ y_2 | x) = \sigma(r(x, y_1) - r(x, y_2))$

where $\sigma$ is the sigmoid function. This is the same logistic model used in Elo ratings for chess — one player's probability of winning depends on the rating gap. When the reward gap is large, the preference is nearly certain; when it's small, it's a coin flip.

DPO replaces the explicit reward $r$ with the implicit reward defined by the policy's log-ratio against the reference. The model that was "just" generating text is secretly assigning reward scores to every response it could produce.

The parameter $\beta$ controls the trade-off between reward maximization and staying close to the reference policy. Think of it as a leash length:

• High β (long leash): the model is free to diverge far from the reference. It can chase the reward aggressively, but risks degenerating — producing text that scores well on the preference data but sounds unnatural.
• Low β (short leash): the model must stay close to the reference. It's more conservative, producing safer but potentially less aligned outputs.

In the DPO loss, $\beta$ appears as a scaling factor on the log-ratios. A larger $\beta$ amplifies the reward margin, making the sigmoid more confident and the gradients sharper. A smaller $\beta$ compresses the margin, making updates gentler. In practice, $\beta \approx 0.1$ to $0.5$ works well for most alignment tasks.

Let's trace what happens for a single training example $(x, y_w, y_l)$:

Step 1. Compute the log-probabilities of both responses under the current policy $\pi_\theta$ and the frozen reference $\pi_{\text{ref}}$.

Step 2. Calculate the log-ratios: $\hat{r}_\theta(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ for both $y_w$ and $y_l$. These are the implicit rewards.

Step 3. Compute the reward margin: $\hat{r}_\theta(x, y_w) - \hat{r}_\theta(x, y_l)$. This is how much the model currently "rewards" the preferred response over the rejected one.

Step 4. Pass the margin through the sigmoid $\sigma$ to get the model's predicted preference probability.

Step 5. Compute the binary cross-entropy loss: $-\log \sigma(\text{margin})$. If the model correctly ranks $y_w$ above $y_l$ by a large margin, the loss is near zero. If it ranks them incorrectly, the loss is high.

Step 6. Backpropagate. The gradient increases $\pi_\theta(y_w|x)$ and decreases $\pi_\theta(y_l|x)$, weighted by the model's current error.

DPO is not an approximation — it is mathematically equivalent to the RLHF objective under the Bradley-Terry preference model. The derivation has three steps:

1. The KL-constrained reward maximization has a closed-form optimal policy (the Gibbs distribution result from statistical mechanics).
2. This optimal policy can be rearranged to express the reward in terms of the policy and the reference.
3. Substituting this "implicit reward" into the Bradley-Terry log-likelihood for preference data yields the DPO loss.

At no step was an approximation made. If the Bradley-Terry model is the correct preference model, the global minimum of the DPO loss is the same policy that the full RLHF pipeline would converge to — with infinite data and perfect optimization.

The practical difference is optimization trajectory: PPO explores by sampling from the current policy, while DPO optimizes on a fixed offline dataset. This makes DPO simpler but means it can't discover reward signals not already captured in the preference data.

Rafailov et al. tested DPO on three tasks:

Controlled sentiment generation — fine-tuning GPT-2 to generate positive movie reviews. DPO matched or exceeded PPO while being simpler to tune.

Summarization — aligning a model on the TL;DR summarization dataset. DPO achieved comparable reward scores to PPO-based approaches with significantly less computational cost.

Single-turn dialogue — training on the Anthropic Helpful and Harmless dataset. DPO produced responses with higher win rates against the SFT baseline than PPO, while requiring no reward model and converging faster.

Across all tasks, DPO showed a consistent pattern: similar or better alignment quality, with dramatically simpler training. The absence of a reward model eliminated reward hacking, and the supervised optimization eliminated PPO's instability.

DPO's conceptual contribution may outlast its technical one. It revealed that reinforcement learning was not essential for alignment — the "RL" in "RLHF" was a choice, not a necessity. The same objective could be optimized with supervised learning, and the reward model was always implicit in the policy. This reframing opened a new design space for alignment algorithms that the field is still exploring.