Reconciling Modern Machine-Learning Practice and the Classical Bias–Variance Trade-Off

Every machine learning textbook tells the same story. You have training data, and you want to find a predictor $h$ from some function class $\mathcal{H}$ that performs well on new, unseen data. The standard approach is Empirical Risk Minimization (ERM): pick the function that minimizes the training loss.

The classical wisdom says: if $\mathcal{H}$ is too small, every function in it underfits — none can capture the true pattern. If $\mathcal{H}$ is too large, the ERM solution overfits — it memorizes noise and spurious patterns. The textbook conclusion: find the sweet spot in between. This produces the famous U-shaped test risk curve, where performance first improves then degrades as model capacity grows.

This thinking guided decades of practice. Regularization, early stopping, cross-validated model selection — all are tools for finding that sweet spot. The unspoken rule was: a model with zero training error is overfit and will generalize poorly.

Then came deep learning. Practitioners discovered that the recipe for success was not finding the sweet spot — it was building networks so large that they could easily memorize every training example with zero loss. This is called interpolation: the model passes through every training point exactly, like a curve drawn through every dot on a scatter plot.

According to classical theory, this should be catastrophic overfitting. The model has memorized the training data including all its noise — surely it will fail on new data. But empirically, these interpolating models achieved state-of-the-art test accuracy. Zhang et al. (2017) demonstrated this dramatically: neural networks that could memorize completely random labels still generalized well when trained on real data.

A best practice emerged in deep learning: the network should be large enough to achieve zero training loss effortlessly. This directly contradicted the textbook rule. Something fundamental was missing from the classical picture.

Belkin et al. proposed a unified framework that resolves the contradiction. The classical U-shaped curve is not wrong — it is incomplete. It only shows what happens in the under-parameterized regime, where the model has fewer parameters than training examples. If you keep increasing model capacity past the interpolation threshold, a second phenomenon emerges.

The full picture is the double descent curve. It has three distinct regions. On the left, the classical regime: underfitting gives way to a sweet spot, then overfitting increases as capacity approaches $n$ (the number of training examples). At the interpolation threshold ($N \approx n$, where $N$ is the number of parameters), the model just barely has enough capacity to fit all training points — and test risk peaks sharply. This is the worst possible model: all its capacity is consumed by the constraint of fitting the data exactly, with no room for any useful inductive bias.

But past this threshold, in the over-parameterized regime ($N \gg n$), something remarkable happens: test risk decreases again. The model has so many parameters that it can fit the training data in many different ways — and among all interpolating solutions, the learning algorithm finds progressively smoother, simpler ones. More capacity means more choices, and more choices means better solutions.

All interpolating models (those to the right of the threshold) have zero training error. So what distinguishes a good interpolator from a bad one? The answer lies in the inductive bias — the implicit preference of the learning algorithm for certain solutions over others.

For Random Fourier Features (RFF), the learning algorithm finds the minimum-norm solution: among all coefficient vectors $(a_1, \ldots, a_N)$ that achieve zero training loss, it picks the one with the smallest $\ell_2$ norm. This is the mathematical formulation of Occam's razor — choose the simplest explanation compatible with the data.

The key mathematical fact: as the number of features $N$ grows beyond $n$, the minimum-norm interpolating function approximates the minimum-norm function in the full Reproducing Kernel Hilbert Space (RKHS). This RKHS-norm solution is the smoothest possible interpolator. More features means a closer approximation to this ideal smooth function.

The same mechanism works across different model families, though the specific inductive bias differs. For neural networks trained with SGD, there is empirical and theoretical evidence that gradient descent implicitly favors low-norm solutions — this is known as implicit regularization. For random forests, the inductive bias comes from averaging: each individual interpolating tree may be rough, but averaging many such trees produces a smooth interpolating function.

Remarkably, for kernel machines, three seemingly different approaches — explicit norm minimization, SGD from zero initialization, and averaging Gaussian process trajectories — all converge to the same minimum-norm interpolating solution. This convergence suggests that the double descent phenomenon is not an artifact of a particular algorithm, but a fundamental property of how model capacity interacts with data.

A central contribution of this paper is showing that double descent is not peculiar to one model type — it appears across fundamentally different learning methods.

Random Fourier Features (RFF): Two-layer neural networks with fixed random first-layer weights. On MNIST ($n=10{,}000$), the test risk curve shows a clean U-shape for $N < n$, peaks sharply at $N = n$, and descends again for $N > n$. The minimum-norm kernel machine solution ($\hat{h}_{n,\infty}$) outperforms every finite-$N$ model. The same pattern repeats on CIFAR-10, SVHN, TIMIT, and 20-Newsgroups.

Fully connected neural networks: Single hidden-layer networks trained with SGD on MNIST. The interpolation threshold appears at $N \approx n \cdot K$ (where $K$ is the number of classes). Despite the non-convexity of the optimization landscape, double descent is clearly visible, though noisier than with RFF.

Random forests and boosted trees: Using the number of leaves per tree and the number of trees as a proxy for capacity, double descent appears. Averaging many interpolating trees produces smoother solutions with lower test risk than any individual tree.

The authors provide a theoretical foundation for why minimum-norm interpolation works. Suppose the data is generated by a target function $h^*$ in the RKHS $\mathcal{H}_\infty$, with no noise. The training points $x_1, \ldots, x_n$ are drawn uniformly from a compact domain $\Omega$. Then any interpolating function $h \in \mathcal{H}_\infty$ satisfies a uniform approximation bound:

If double descent is so universal, why did the field miss it for decades? The authors identify several practical and cultural reasons.

First, classical statistics typically works with fixed, small feature sets — there is no way to increase capacity past the interpolation threshold. Second, in non-parametric statistics, regularization is almost always used, and regularization masks the interpolation peak by preventing exact fitting. Third, RFF models were originally proposed as a cheap alternative to kernel machines, so they were typically used with $N \ll n$ — well below the interpolation threshold.

For neural networks specifically, the non-convexity of training makes the curve noisy and hard to observe. Early stopping — stopping training when validation error starts rising — has a strong regularizing effect that hides the peak. And the peak itself occurs in a narrow range of parameters that is easy to miss when sampling architectures coarsely.

The double descent framework has three immediate practical consequences.

First, model selection changes. The classical rule "pick the sweet spot" only applies in the under-parameterized regime. In modern practice, if you can afford to train a model past the interpolation threshold, making it larger improves generalization. This gives theoretical backing to the empirical deep learning practice of using very large networks.

Second, optimization becomes easier. Over-parameterized models have favorable optimization landscapes: SGD converges to global minima more reliably when $N \gg n$. So larger models are both easier to train and better at generalization — a double benefit that classical theory never predicted.

Third, regularization should be reconsidered. While regularization helps in the classical regime, it can prevent a model from reaching the beneficial over-parameterized regime by keeping it from interpolating. The paper suggests that practitioners should be more careful about when and how much regularization to apply.

The double descent curve fundamentally changed how the field thinks about the relationship between model size and generalization. Before this paper, "bigger model" was associated with "more overfitting." After it, the community understood that there are two regimes — and the modern regime, where larger models find smoother interpolations, explains the empirical success of deep learning. This insight is now woven into the foundation of the scaling-laws era, where the guiding principle is: if resources allow, make the model bigger.