Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Today's alignment pipeline works because humans can judge model outputs. When ChatGPT writes a paragraph, a human can tell if it is helpful, accurate, and safe. RLHF works precisely because human evaluators are competent supervisors.

But what happens when models become superhuman? If a model writes a million lines of code, designs a novel protein, or produces a mathematical proof that spans hundreds of pages, humans can no longer provide reliable supervision. The human becomes a weak supervisor — they can catch surface-level errors but miss deeper failures.

This creates a fundamental tension: the very technique we use to align models (human feedback) breaks down exactly when alignment matters most (superhuman capabilities). The authors call this the superalignment problem: how can weak supervisors control models much smarter than them?

The authors propose a clean three-step experimental framework. First, finetune a small model on ground truth labels to create a weak supervisor. Second, use that weak supervisor's predictions as training labels to finetune a much larger strong student model. Third, finetune the same strong model on ground truth labels to establish a strong ceiling — the best the strong model can do with perfect supervision.

The key metric is the Performance Gap Recovered (PGR): the fraction of the gap between weak performance and strong ceiling that the weak-to-strong student manages to close. A PGR of 0 means the student learned nothing beyond what the weak supervisor knows. A PGR of 1 means perfect generalization — the student recovered everything the strong model is capable of, despite only seeing weak labels.

The beauty of this setup is that it can be studied today with any pair of models — no superhuman AI required. It removes a key bottleneck: we no longer need to wait for superhuman models to study the alignment challenges they will create.

The authors tested their framework across three domains using pretrained models from the GPT-4 family, spanning 7 orders of magnitude of compute.

NLP benchmarks (22 tasks): The strongest results. When GPT-4 is finetuned on labels from a GPT-2-level supervisor, the PGR often exceeds 50%. The strong model consistently outperforms its weak supervisor by a wide margin — it doesn't just copy the weak labels; it leverages its own superior representations.

Chess puzzles: More mixed. Small supervisor-student gaps yield decent PGR (above 40%), but for very large gaps the PGR drops and even shows signs of inverse scaling — larger students sometimes generalize worse.

ChatGPT reward modeling: The hardest setting. PGR is typically around 10%, rarely exceeding 20%. This suggests that naive RLHF will scale poorly to superhuman models without new methods.

A critical finding across all three domains: PGR is almost universally positive. In nearly every configuration, the strong student outperforms its weak supervisor. The strong model doesn't merely copy — it generalizes. This is the encouraging core result.

Naive finetuning gives positive PGR, but falls short of recovering the strong model's full capability. The authors tested three simple methods to close this gap.

The most successful method. The core problem with naive finetuning is that the student is trained to imitate the weak supervisor — including its errors. We want the student to learn the supervisor's intent, not its mistakes.

The solution: add an auxiliary loss term that encourages the strong model to be confident in its own predictions, even when they disagree with the weak labels. This is closely related to entropy minimization in semi-supervised learning. The intuition is that the strong model's own confident predictions are more likely to be correct than the weak labels where they disagree, because the strong model has better internal representations.

The confidence loss adds a term to the training objective that rewards the strong student for making high-confidence predictions on unlabeled data. This regularizes the student away from blindly imitating weak labels and toward its own internal knowledge.

With this simple addition, PGR on NLP tasks jumps from about 25% to nearly 80% when using the smallest weak supervisor and largest strong student. On many individual tasks, the method recovers almost all of the strong model's performance.

Instead of one giant leap from weak to strong, take many small steps. Train a slightly-stronger model on the weak labels, use it to generate better labels, and repeat — climbing through a sequence of models like ascending a staircase.

Formally: construct a chain $\mathcal{M}_1 \to \mathcal{M}_2 \to \cdots \to \mathcal{M}_n$ of increasing sizes. Use weak labels from $\mathcal{M}_1$ to finetune $\mathcal{M}_2$, use $\mathcal{M}_2$'s predictions to finetune $\mathcal{M}_3$, and so on.

This works well for chess puzzles, where naive PGR decreases with larger gaps. Since each small step has high PGR, chaining many small steps preserves generalization. With bootstrapping, accuracy continues to improve monotonically with model size, fixing the flat scaling observed with naive transfer. On NLP tasks, where naive PGR already scales well, bootstrapping provides only modest additional gains.

If weak-to-strong generalization works better when the task is "salient" — when the strong model has strong internal representations of the relevant concepts — then we can improve generalization by increasing saliency before doing weak-to-strong transfer.

The idea is to first finetune the strong model with a standard language modeling loss on task-relevant data, without using any labels. For reward modeling, this means finetuning on all prefix-completion pairs from the ChatGPT comparison dataset, ignoring which completion humans preferred.

This unsupervised step makes the concepts relevant to the task (like "what makes a good assistant response") more linearly represented in the model's activations. When weak-to-strong training follows, the weak supervisor's signal finds a better-prepared landscape. This boosts PGR by 10–20% in the difficult reward modeling setting.

Why doesn't the strong model always reach its full potential? The main failure mode is imitation: the student learns to reproduce the weak supervisor's predictions — including its systematic errors — rather than learning the underlying task.

Three pieces of evidence reveal this failure mode:

Overfitting to weak labels. Ground truth test accuracy initially rises during training, then drops — even within a single epoch. The strong model starts by leveraging its own knowledge, but gradually learns to imitate the weak supervisor's errors. This "weak label overfitting" is distinct from standard overfitting to training examples.

High student-supervisor agreement. When measured on test data, the strong student agrees with the weak supervisor more often than the weak supervisor is correct. This means the student is copying some of the supervisor's mistakes.

Inverse scaling for imitation. The surprising finding: larger strong models agree less with their supervisor's errors. Despite having more capacity to memorize weak labels, larger pretrained models seem to have a harder time fitting the error patterns of other models. This suggests that pretrained representations actively resist learning incorrect patterns — an encouraging sign for scaling.

Weak-to-strong generalization works best when the desired task is internally "salient" to the strong model — meaning the relevant concepts are well-represented in the model's activations.

Prompting reveals saliency. For the largest models, zero-shot and 5-shot prompting can nearly match finetuning on many NLP tasks, indicating that the task knowledge is easily accessible. Even few-shot prompting with weak labels shows weak-to-strong generalization similar to finetuning, suggesting the phenomenon is not an artifact of the training procedure.

Weak finetuning linearizes representations. A fascinating discovery: finetuning on weak labels makes the ground truth concept more linearly represented in the model's activations. A linear probe trained on ground truth labels after weak finetuning significantly outperforms one trained on the base model. This means weak supervision reorganizes the model's internal representations in a way that actually helps, even though the labels themselves are imperfect.

This insight connects to findings in the spurious correlations literature: finetuning on biased data can produce biased outputs, but surprisingly strong representations of the correct concept.

The authors are careful to note important limitations of their framework:

Imitation saliency. Future superhuman models will be explicitly trained on human data and will likely have very salient representations of human- level behavior. This makes the "human simulator" failure mode — where the model learns to output what a human would say rather than its own best answer — much more dangerous than in the current setup, where strong models are not pretrained to imitate weak models.

Pretraining leakage. Many NLP tasks studied here may appear indirectly in the pretraining data. This could make eliciting capabilities artificially easy today. Superhuman capabilities may be more "latent" — learned through self-supervised learning rather than directly observed — and thus harder to elicit.

Despite these caveats, the authors conducted a proof-of-concept using self-supervised vision models (DINO) where capabilities are latent and prompting is impossible. Even there, weak-to-strong generalization succeeded, giving reason for cautious optimism.

The paper concludes with a call to action: superhuman models may arrive this decade, and aligning them is among the most important unsolved technical problems. The weak-to-strong framework provides a way to make empirical progress now — before superhuman models exist — on the fundamental challenge of ensuring they do what we want.