Emergent Abilities of Large Language Models

The concept of emergence comes from physics. When water molecules are heated, each molecule moves a little faster — a smooth, predictable change. But at 100°C, the system undergoes a phase transition: liquid becomes gas. The collective behavior is qualitatively different, even though the underlying change (more kinetic energy) was gradual.

Wei et al. borrow this idea for language models. They define an emergent ability as one that is "not present in smaller models but is present in larger models." Crucially, this means the ability cannot be predicted by extrapolating the performance curve of smaller models. If you plotted performance against scale for a non-emergent ability, you would see a smooth upward curve. For an emergent ability, you see a flat line at random-chance performance, then a sharp jump.

The authors measure scale along two axes: the number of model parameters and the total training compute in FLOPs (floating-point operations). Using FLOPs is important because it captures both model size and how long the model was trained — a smaller model trained longer can match a larger model trained less.

The first category of emergence the paper examines is few-shot prompting. In this paradigm, popularized by GPT-3, you give the model a natural language instruction plus a handful of input-output examples, then ask it to perform the task on a new input — with no gradient updates or fine-tuning.

The key finding: for many tasks, small models perform at or near random chance no matter how good the prompt is. Then, above a critical scale, performance jumps sharply. The paper documents this pattern across diverse tasks from two major benchmarks.

BIG-Bench — a crowd-sourced suite of over 200 tasks — yielded striking examples. Three-digit addition and subtraction shows near-zero accuracy for GPT-3 below 13B parameters ($\sim 2 \times 10^{22}$ FLOPs), then jumps to high accuracy. LaMDA shows the same pattern but at 68B parameters ($\sim 10^{23}$ FLOPs). Other emergent BIG-Bench tasks include transliterating from the International Phonetic Alphabet, recovering scrambled words, and Persian question-answering.

MMLU (Massive Multitask Language Understanding) — a 57-subject benchmark — showed that different subject areas become emergent at different scales. Humanities and Social Sciences were the most emergent categories, appearing above random only at the largest model sizes. The pattern was consistent across GPT-3, Chinchilla, and Gopher.

A third benchmark, TruthfulQA, revealed an initially counterintuitive pattern: larger Gopher models actually performed worse on truthfulness, generating more confident but incorrect answers. Only at the very largest scale did performance recover. This suggests that emergence is not always a simple story of "bigger is better" — some abilities may temporarily degrade during the scaling process before ultimately emerging.

The second category goes beyond individual tasks. Some prompting strategies themselves are emergent — they hurt small models but help large ones.

Chain-of-thought prompting asks the model to show its reasoning step by step before giving a final answer. For the LaMDA model family, chain-of-thought prompting only surpasses standard prompting above $\sim 10^{23}$ FLOPs (roughly 100B parameters). Below that threshold, asking the model to reason step-by-step actually hurts performance compared to direct answering. The model is too small to maintain coherent multi-step reasoning, so the extra steps introduce errors rather than helping.

Instruction following shows a similar pattern. Finetuning on a mixture of tasks phrased as instructions (the FLAN approach) enables zero-shot generalization to unseen tasks — but only above about 68B parameters. Below that scale, instruction finetuning actually degrades performance relative to the base model.

Scratchpad training, where the model writes out intermediate computation steps (like carrying digits in addition), only helps larger models. For smaller LaMDA models, the scratchpad adds noise rather than structure.

The paper is deliberately open about this: we do not know. The authors present the empirical pattern but do not provide a mechanistic explanation. They offer several hypotheses worth considering.

One possibility is that emergent abilities require a constellation of sub-skills. A model needs to simultaneously master tokenization, syntax, semantics, world knowledge, and task format — and only when all of these cross individual quality thresholds does the combined ability appear. Think of it like a combination lock: each dial moving closer to the right number does nothing visible until all dials click into place at once.

Another hypothesis involves the relationship between pretraining loss and task performance. While loss decreases smoothly, the mapping from loss to performance on a specific task may be highly nonlinear. A small improvement in loss near a critical region could correspond to a large jump in task accuracy — especially for tasks with sharp success criteria like exact-match accuracy.

The paper does not resolve which explanation is correct. This open question became one of the most active research areas following publication.

In 2023, Schaeffer, Miranda, and Koyejo published a provocative response: Are Emergent Abilities of Large Language Models a Mirage? Their argument is elegant and important.

They observed that almost all "emergent" tasks were measured with discontinuous metrics — metrics like exact-match accuracy that give zero credit for partial answers. Under such metrics, a model that gets 4 out of 5 digits right in a math problem scores identically to a model that guesses randomly. The result: apparent flat-line performance even though the model is improving steadily.

When Schaeffer et al. re-evaluated the same tasks with continuous metrics (like token-level edit distance or per-digit accuracy), the sharp transitions vanished. Performance improved smoothly and predictably with scale. Over 92% of emergent abilities on BIG-Bench disappeared when evaluated with linear metrics.

Their conclusion: emergence may not be a property of the model but a property of the measurement. The model's underlying capability improves smoothly, but a harsh metric creates the illusion of a sudden jump — like measuring a high-jumper's ability as pass/fail on a fixed bar height rather than tracking how high they actually jump.

Whether emergence is fundamental or partly a measurement artifact, the practical implications are profound.

For model developers: if truly emergent abilities exist, they cannot be predicted by evaluating smaller models. This means you cannot fully characterize a model's capabilities until it is built and tested — a deeply uncomfortable position for safety planning. The only way to know what a 10-trillion-parameter model can do is to build it.

For AI safety: unpredictable capability acquisition is a core concern. If models can gain abilities that were not anticipated, they could also gain dangerous capabilities — like generating convincing misinformation or exploiting security vulnerabilities — without warning. The paper explicitly notes that emergence could include harmful abilities.

For benchmarks: the debate about metrics reveals that how we measure model capabilities matters as much as what we measure. Coarse binary metrics may hide gradual improvement, while fine-grained metrics may miss qualitative transitions. The field needs evaluation frameworks that capture both.

The paper closes with an observation that doubles as a prediction: "Further scaling of language models could potentially expand the range of capabilities even further." As of 2024 and beyond, that prediction has been borne out repeatedly.

The emergence debate illuminated a deeper truth about AI research: what we observe depends on how we measure. Whether emergent abilities are phase transitions or metric artifacts, the practical reality is the same — scaling language models produces capabilities that were not anticipated, cannot be easily predicted, and demand careful evaluation. The paper's lasting contribution is not a definitive answer about emergence but a framework for asking the right questions about what scaling does and does not tell us.