Optimal Brain Damage

By the late 1980s, neural networks were achieving strong results on real-world tasks like handwritten digit recognition. But practitioners faced a fundamental tension: a network with too many parameters memorizes the training data and generalizes poorly (overfitting), while a network with too few parameters cannot capture the underlying patterns (underfitting).

The ideal solution is to find the right-sized network — one with just enough parameters to model the data, and no more. But how do you find that sweet spot? You could train many networks of different sizes and compare them, but that is prohibitively expensive. LeCun, Denker, and Solla proposed a far more elegant approach: start with a network that is too large, then surgically remove the parameters that contribute the least.

The most obvious pruning strategy is magnitude pruning: rank all weights by their absolute value and delete the smallest ones. The intuition is simple — a weight near zero contributes little to the output, so removing it should not matter much.

But this intuition is dangerously incomplete. Consider a weight that is small in value but sits on a steep slope of the loss surface. Removing it causes the loss to increase sharply — like removing a small but critical bolt from a bridge. Conversely, a weight that is large but sits in a flat valley of the loss surface can be removed with minimal effect — like removing a heavy but redundant support beam.

The key insight of OBD is that what matters is not the weight's value, but how sensitive the loss is to changes in that weight. This sensitivity is captured by the second derivative — the curvature of the loss surface.

OBD defines the saliency of a parameter as the predicted increase in the loss function when that parameter is set to zero. To compute this without actually deleting each weight and retraining, OBD uses a second-order Taylor expansion of the loss function around the trained weights.

The Taylor expansion of the change in loss $\delta E$ when the parameter vector is perturbed by $\delta u$ is:

$\delta E = \sum_i g_i \delta u_i + \frac{1}{2}\sum_i h_{ii} \delta u_i^2 + \frac{1}{2}\sum_{i \neq j} h_{ij} \delta u_i \delta u_j + O(\|\delta u\|^3)$

where $g_i = \frac{\partial E}{\partial u_i}$ is the gradient and $h_{ij} = \frac{\partial^2 E}{\partial u_i \partial u_j}$ are the entries of the Hessian matrix.

Think of each weight as a ball sitting on a curved surface. The saliency formula asks two things: how far is the ball from the bottom of the valley? (the weight value $u_k$) and how steep are the valley walls? (the curvature $h_{kk}$). A ball far up a steep slope will cause a big drop if removed — high saliency. A ball barely off the valley floor, or on a flat plain, barely matters — low saliency.

The Hessian matrix for a network with $N$ parameters has $N^2$ entries — for a 2600-parameter network, that is over 6.5 million numbers. Computing the full Hessian is impractical. But OBD only needs the diagonal entries $h_{kk}$, which can be computed with a procedure almost identical to standard backpropagation.

The key observation is that second derivatives propagate backward through the network just like first derivatives. Starting from the output layer, where the second derivative of the loss with respect to the pre-activation is known, we can recursively compute the second derivative at each layer using the chain rule — the same machinery that computes gradients, applied one level deeper.

The cost of computing the diagonal Hessian is roughly the same as computing the gradient — one additional backward pass. This made OBD practical even on the hardware of 1989.

The OBD procedure alternates between training and pruning in a cycle:

Step 1 — Train the network to convergence on the training data, using standard backpropagation.

Step 2 — Compute the diagonal second derivatives $h_{kk}$ for every parameter, using the backward pass described above.

Step 3 — Compute the saliency $s_k = h_{kk} u_k^2 / 2$ for each parameter.

Step 4 — Sort all parameters by saliency and delete the lowest-saliency ones (set them to zero and freeze them).

Step 5 — Retrain the pruned network and repeat from Step 2.

"Deleting" a parameter means setting it to zero and freezing it — it can never come back. Each cycle removes a batch of low-saliency parameters. The process stops when further pruning would significantly degrade performance.

LeCun et al. tested OBD on their handwritten digit recognition network — a constrained, sparsely connected architecture with about 105,000 connections controlled by 2,578 free parameters, trained on ~9,300 examples of segmented zip-code digits.

The results were striking. OBD could remove over 60% of parameters (from 2,578 down to ~1,000) with essentially no degradation in test accuracy — and sometimes the pruned network actually generalized better than the original. When used interactively, OBD revealed that an entire layer of the network was largely redundant, enabling the authors to redesign the architecture and halve the parameter count.

Crucially, OBD significantly outperformed magnitude-based pruning. At the same compression ratio, magnitude pruning caused much larger increases in the objective function. Random deletion was so destructive that its results could not even be plotted on the same scale.

OBD's elegance rests on its three simplifying assumptions, but each introduces limitations:

The diagonal approximation ignores interactions between weights. In reality, deleting one weight changes the optimal values of others. The follow-up paper Optimal Brain Surgeon (Hassibi & Stork, 1992) addressed this by using the full Hessian inverse, allowing remaining weights to compensate for the deleted one.

The extremal approximation assumes training has converged perfectly. In practice, gradient norms may not be exactly zero, especially with early stopping or noisy data.

The quadratic approximation breaks down when large weights are deleted — the loss surface is not parabolic far from the minimum. The paper's own experiments showed the prediction diverging from reality after ~30% of parameters were removed in a single pass.

Despite these limitations, OBD demonstrated that even an approximate second-order method dramatically outperforms magnitude-based heuristics.

OBD's core question — which parameters can be safely removed? — is more relevant today than it was in 1989. Modern large language models have billions or trillions of parameters, making compression essential for deployment on edge devices, reducing inference costs, and lowering energy consumption.

The intellectual lineage from OBD to modern methods is direct. Deep Compression (Han et al., 2015) combined magnitude pruning with quantization and Huffman coding to compress networks by 35–49×. The Lottery Ticket Hypothesis (Frankle & Carlin, 2019) showed that dense networks contain sparse subnetworks that can match the full network's accuracy when trained in isolation — echoing OBD's discovery that most parameters are redundant. GPTQ (Frantar et al., 2022) applied second-order information (in the spirit of OBD and OBS) to quantize large language models with minimal accuracy loss. And SparseGPT extended this to pruning LLMs to 50–60% sparsity in a single shot.

The fundamental principle OBD established — use curvature information to measure parameter importance, not just magnitude — remains the gold standard in network compression. Every time a modern LLM is quantized or pruned for deployment, it carries the intellectual DNA of this 1989 paper.