Scaling Laws for Neural Language Models

When you train a language model, three resources determine how good it gets:

Model size (N) — the number of learnable parameters (weights) in the network, excluding embeddings. A 100M-parameter model is a small brain; a 175B-parameter model is a vastly larger one.

Dataset size (D) — how many tokens the model sees during training. More tokens means more patterns to learn from.

Compute (C) — the total floating-point operations spent on training, roughly $C \approx 6NBS$ where $B$ is the batch size and $S$ is the number of training steps.

The breakthrough insight is that these three variables are all you need. Architectural details like depth vs. width, number of attention heads, or feed-forward dimension barely matter once you hold the total parameter count fixed.

The paper's central discovery is that when you plot loss against any of the three variables on a log-log scale, you get a straight line. A straight line on a log-log plot means a power law — the relationship $L \propto X^{-\alpha}$, where $\alpha$ is the slope.

Think of it this way: every time you multiply the model size by 10, the loss drops by a fixed percentage. Not a fixed amount — a fixed fraction. This pattern holds whether you go from 1,000 parameters to 10,000 or from 100 million to 1 billion — the same proportional improvement each time.

Why does this matter? Because power laws are predictable. If you've measured the trend on small models, you can extrapolate to large ones with confidence. Training a small model is cheap; predicting how a large model will perform before you build it is invaluable.

The paper fits three core power laws. Each describes what happens when one variable is the bottleneck and the other two are abundant. Before seeing the formulas, here is the intuition: imagine filling a bathtub. The water level (performance) is limited by whichever pipe (parameter count, data, compute) carries the least flow. The power law tells you how much the water rises when you widen that pipe.

One of the most surprising findings: when you fix the total parameter count, the model's shape barely affects performance. The ratio of depth to width can vary by a factor of 40 — from a very deep, narrow model to a very wide, shallow one — with only a 3% change in loss.

This was a radical claim. Before this paper, architecture search was a major research area. The paper suggested that all that effort was optimizing the wrong thing: instead of finding the perfect shape, you should simply be making models bigger.

The one exception: excluding embedding parameters from the count N gives much cleaner trends. This suggests embeddings are somewhat "free" capacity that doesn't contribute to the core scaling behavior.

What happens when you make the model larger but keep the dataset fixed? Eventually the model memorizes the data instead of learning patterns — this is overfitting.

The paper discovered a remarkably clean relationship: the degree of overfitting depends on a single ratio $N^{0.74}/D$. This means that every time you make the model 8× bigger, you only need roughly 5× more data to avoid performance penalties. Data requirements grow sub-linearly with model size.

This is great news for scaling: you don't need to proportionally increase your dataset every time you build a bigger model. The model becomes increasingly sample-efficient — it extracts more knowledge from each token.

This is the paper's most practically important result. Given a fixed compute budget $C$, how should you distribute it between a bigger model and more training time?

The answer is dramatic: most of the compute should go to model size. As compute increases by 1000×, the optimal model size grows by about 500× (exponent 0.73), while the number of training steps barely increases (exponent 0.03). The data requirement grows slowly too — only about 7× (exponent 0.27).

In practical terms, this means compute-efficient training involves training very large models for a short time, stopping well before convergence. This is counterintuitive: most researchers in 2020 were training smaller models to full convergence. The paper argued they were wasting compute.

A consistently striking finding: larger models are dramatically more sample-efficient. A 1-billion parameter model reaches the same loss as a 10-million parameter model while seeing roughly 100× fewer data examples.

This creates a virtuous cycle for scaling: not only do larger models converge to lower loss, they also get there faster in terms of data. The optimal strategy is not "collect as much data as possible and train a big model on all of it," but rather "build the biggest model you can afford and train it on a modest amount of data."

The learning curves themselves follow a power law: $L(N, S) = (N_c/N)^{\alpha_N} + (S_c/S)^{\alpha_S}$ where $S$ is the number of training steps. This means you can predict the final loss from the first few percent of training — a powerful tool for deciding whether to continue an expensive run.

There is an optimal batch size for training, and it too follows a power law — but remarkably, it depends only on the loss value, not on the model size. The critical batch size roughly doubles for every 13% decrease in loss, following $B_{crit} \approx B^*/L^{1/\alpha_B}$ with $\alpha_B \approx 0.21$.

Training below the critical batch size wastes time (too many serial steps). Training above it wastes compute (each step gives diminishing returns). The sweet spot, training at the critical batch size, makes the best use of both.

For the largest models approaching convergence, the critical batch size is roughly 1–2 million tokens — which maps to the large-batch training configurations that became standard practice in subsequent work.

The paper includes a crucial comparison between Transformers and LSTMs. Both follow power-law scaling with parameters, but Transformers asymptotically outperform LSTMs — and the gap widens as models grow.

The reason is context utilization. LSTMs plateau after about 100 tokens of context: adding more context doesn't help. Transformers, by contrast, continue to improve throughout their entire 1024-token context window. Each additional word of context makes the Transformer's predictions meaningfully better.

This finding provided quantitative evidence for what the Transformer paper had argued qualitatively: self-attention's direct word-to-word connections are not just architecturally elegant — they are the mechanism that enables scaling.

The authors noticed a deep tension in their own equations. Compute-efficient training grows data very slowly — $D \propto C^{0.27}$. But avoiding overfitting requires $D \propto N^{0.74} \propto C^{0.54}$. These two trends eventually contradict: at some point, compute-efficient training won't have enough data to avoid overfitting, even if you never reuse a single token.

The authors estimated this intersection at roughly $10^{12}$ parameters and $10^{12}$ tokens, with a loss of about 1.7 nats/token. They conjectured this might represent the inherent entropy of natural language — an irreducible floor below which no model can go, no matter how large.

This prediction was remarkably prescient. Modern frontier models have indeed approached the trillion-parameter, trillion-token scale, and the quest for data has become one of the field's most pressing challenges.

When the authors tested their WebText2-trained models on completely different text distributions — Wikipedia, Books, Common Crawl — they found a consistent pattern: the loss on other distributions follows the same power law with model size, just shifted by a constant offset.

This means that improving on the training distribution translates directly to improvement on other distributions. A model doesn't just memorize WebText2 patterns — it learns general language understanding that transfers. Furthermore, this generalization gap doesn't depend on training duration or model depth; it depends only on the validation loss achieved.

Kaplan et al. didn't just discover power laws — they created a predictive science of AI scaling. Before this paper, training a frontier model was an act of faith. After it, teams could write down equations, plug in their compute budget, and predict within a few percent what performance they would achieve. The specific numbers have been refined — the Chinchilla paper corrected the data/model balance — but the paradigm of "fit power laws on small runs, extrapolate to large ones" is now standard practice at every AI lab.