WebGPT: Browser-Assisted Question-Answering with Human Feedback

The core innovation is deceptively simple: instead of trying to improve document retrieval with a new neural architecture, the authors gave GPT-3 access to a familiar, powerful existing tool — the Bing search engine — through a text-based interface. The model sees a text summary of the current browser state: the question, the visible portion of the current web page, a scrollbar position indicator, and a list of past actions. It then issues one of a fixed set of commands — search, click a link, scroll, quote a passage, or end browsing — and the cycle repeats.

Think of it as a command-line browser: there are no images, no JavaScript rendering, just cleaned-up text from each page. The model has no memory between steps — the only record of what it has done lives in a compact summary appended to each new prompt. This stateless design is critical: it means the entire browsing session can be represented as a sequence of text completions, making it compatible with standard language model fine-tuning.

While browsing, the model can quote passages from the current page. Each quote is saved with its page title and URL as a reference. When the model finishes browsing, it is prompted with the question and all collected quotes, and composes a final answer with inline citations. This two-phase design — browse then answer — forces the model to ground every claim in evidence it found.

WebGPT's training follows a four-stage pipeline, each building on the previous one. The key insight from Learning to Summarize is that imitating human behavior (behavior cloning) gets you a working baseline, but optimizing for human preferences using a reward model takes you further — potentially beyond human-level performance.

Stage 1 — Behavior Cloning (BC). Human demonstrators use a graphical browser interface to answer questions from the ELI5 dataset. They search, navigate, quote references, and compose answers. Around 6,000 of these demonstrations are collected. GPT-3 is then fine-tuned on these demonstrations using supervised learning, with the human's commands as labels. This gives the model a working ability to use the browser.

Stage 2 — Reward Modeling (RM). The BC model generates pairs of answers to the same question. Human labelers compare each pair and indicate which answer is better based on factual accuracy, coherence, and usefulness. Around 21,500 comparisons are collected. A reward model is trained to predict these human preferences, outputting a scalar score that represents an Elo-style quality rating.

Stage 3 — Reinforcement Learning (RL). The BC model is further fine-tuned using PPO (Proximal Policy Optimization), with the reward model providing the environment reward at the end of each browsing episode. A KL penalty from the BC model is added at each token to prevent the policy from drifting too far and overoptimizing the reward model.

Stage 4 — Rejection Sampling (Best-of-n). Multiple answers are sampled from the BC or RL model, and the one scoring highest on the reward model is selected. This is a simple but powerful alternative to RL that requires no additional training — only more inference-time compute. The best configuration samples 64 answers and picks the best one.

The reward model is the engine that drives quality beyond imitation. It takes a question, an answer, and the collected references as input, and outputs a single scalar — an Elo score that represents how much a human labeler would prefer this answer.

Training uses the comparison data: given two answers A and B to the same question, the reward model learns to assign a higher score to whichever answer the human preferred. The difference between two reward scores represents the log-odds of one being preferred over the other. If answer A has a reward 1 point higher than answer B, then A would be preferred about 73% of the time.

The reward model is initialized from the BC model with the final unembedding layer removed, meaning it inherits the same reading comprehension and language understanding. It is then trained with a cross-entropy loss on the comparison labels, with ties treated as soft 50% labels. The labelers evaluated answers on multiple dimensions — unsupported claims, relevance, coherence, citation quality — but only the final overall preference was used for training, keeping the objective simple and robust.

One of the most striking findings in the paper is that rejection sampling — simply generating many answers and picking the best one according to the reward model — works better than reinforcement learning. The 175B best-of-64 model (behavior cloning + rejection sampling) is preferred 68% of the time over the plain BC model, while RL alone gets only 58%.

Why does this outperform RL? The authors offer several explanations. First, rejection sampling gives the model many independent browsing attempts — it can visit different websites each time and evaluate the information with hindsight. Second, the reward model was trained primarily on data from BC and rejection sampling policies, making it more robust to overoptimization by rejection sampling than by RL. Third, RL requires careful hyperparameter tuning, while rejection sampling is essentially tuning-free.

Think of it as a job interview: RL is like coaching one candidate to give better answers, while rejection sampling is like interviewing 64 candidates and hiring the best one. When the "interviewer" (reward model) is reliable, selecting the best from a large pool can outperform intensive coaching of a single candidate.

The paper evaluates WebGPT in three ways, each revealing a different aspect of its capabilities.

ELI5 vs. human demonstrators. The 175B best-of-64 model is preferred 56% of the time over answers written by the human demonstrators who trained it. This is a meaningful threshold: you would not expect to exceed 50% by imitation alone, so the reward model optimization is clearly providing additional value.

ELI5 vs. Reddit. The same model is preferred 69% of the time over Reddit's highest-voted answers. For fairness, citations were stripped from the model's answers for this comparison, and new labelers with minimal instructions were used. Even without citation formatting advantages, the model substantially outperformed community-generated answers.

TruthfulQA. On an adversarially constructed dataset of short-form questions designed to elicit common misconceptions, WebGPT answered truthfully 75% of the time and was both truthful and informative 54% of the time — substantially better than base GPT-3, though still below human performance. Importantly, truthfulness increased with model size for WebGPT, unlike for the base GPT-3 model where larger models tend to produce more confident falsehoods.

The paper introduces a useful distinction between two categories of false statements that language models make.

Imitative falsehoods are false claims that the model is incentivized to produce by its training objective — even with infinite data and compute. For example, reproducing common misconceptions that appear frequently in the training data. WebGPT reduces these because it can look up facts rather than relying on memorized patterns, and because it is incentivized to prefer reliable sources.

Non-imitative falsehoods are false claims that result from the model failing at its training objective — most notably hallucinations, where the model generates plausible- sounding but fabricated information. WebGPT reduces these because retrieval-augmented generation inherently reduces hallucination, and because the model must ground its answer in collected references.

However, neither category is eliminated. On out-of-distribution questions (like TruthfulQA's adversarial prompts), WebGPT sometimes quotes from highly unreliable sources. And while non-imitative falsehoods are reduced, the model still occasionally makes mistakes when paraphrasing or synthesizing information from its references.

The paper provides detailed scaling analysis across three dimensions.

For data scaling, doubling the number of demonstrations increased the policy's reward model score by about 0.13, and doubling the number of comparisons increased the reward model accuracy by about 1.8%. Returns are diminishing but far from exhausted.

For parameter scaling, doubling the number of parameters in the policy increased its reward model score by roughly 0.09, and doubling the reward model parameters increased accuracy by roughly 0.4%. The trends are noisier than data scaling.

For rejection sampling scaling, the authors analyzed compute-efficient trade-offs between model size and number of samples. The Pareto frontier favors some rejection sampling at every model size: the 760M best-of-4, 13B best-of-16, and 175B best-of-64 configurations are the compute-efficient models. Using too many samples provides diminishing returns because the reward model can be overoptimized.

The paper raises several important concerns about deploying web-browsing language models.

Bias reinforcement. WebGPT inherits GPT-3's biases and amplifies them through search and synthesis. The model tends to accept the implicit assumptions of questions, which could reinforce users' confirmation bias. When asked "What does a wedding look like?" it overwhelmingly assumes a Western, American perspective.

Live web access risks. While WebGPT's actions are limited to Bing searches and link following, the authors note that more capable models with web access could potentially exploit real-world side effects — like editing Wikipedia to create reliable-looking references. They argue that as model capabilities increase, so should the safety burden for granting web access.

Reference cherry-picking. The model is incentivized to find references that labelers will find convincing, not references that reflect a fair assessment of evidence. This is analogous to a lawyer building a one-sided case. The authors suggest methods like debate — where models are trained to find evidence both for and against claims — as a mitigation.

Question stance sensitivity. When questions were phrased to affirm misconceptions (e.g., "Why did the government fake the moon landing?"), the model was more likely to produce inaccurate answers that reinforce the false belief, compared to neutrally or skeptically framed versions of the same question.

WebGPT stands at a pivotal junction in AI history. Looking backward, it drew on GPT-3's language understanding and Learning to Summarize's RLHF pipeline. Looking forward, it pioneered the tool-use paradigm that defines modern AI assistants: the idea that a language model should not try to know everything, but should know how to find everything and present it with verifiable evidence.