When Does Label Smoothing Help?

In standard classification, we train neural networks using hard targets: for a cat image, the target vector is $[1, 0, 0, \ldots, 0]$ — full confidence in the correct class, zero everywhere else. This is one-hot encoding. The network minimizes cross-entropy between these targets and its softmax outputs.

The problem is that hard targets push the network toward extreme confidence. To make the predicted probability of the correct class approach 1, the network must drive the corresponding logit toward infinity. This leads to overfitting: the network memorizes training examples rather than learning generalizable patterns, and its confidence becomes poorly calibrated — it says "99.9% cat" even when the evidence is ambiguous.

Label smoothing, introduced by Szegedy et al. in the Inception-v2 architecture, offers a simple fix. Instead of a hard target $y_k$, we use a smoothed target $y_k^{LS}$ that mixes the hard target with a uniform distribution. With a smoothing parameter $\alpha$, the correct class gets $1 - \alpha + \alpha/K$ and each incorrect class gets $\alpha/K$, where $K$ is the total number of classes.

The paper's most striking finding is a geometric one. When a network is trained with hard targets, the penultimate layer representations form broad, overlapping clouds for each class. Examples from the same class can sit at very different distances from other class templates, preserving rich information about inter-class similarity.

With label smoothing, the picture changes dramatically. The smoothed targets encourage every example to be equally distant from all incorrect class templates. The result is that representations collapse into tight, compact clusters arranged in a regular geometric pattern — like vertices of an equilateral triangle (for 3 classes projected onto 2D).

Think of it as the difference between a parking lot where cars scatter freely within each section versus one with assigned spots — every car in its exact position, equidistant from neighboring sections. The parking lot is more orderly, but you lose the information about which cars were "almost" in the wrong section.

To visualize this, the authors proposed a novel technique. They pick three classes, find the 2D plane passing through the three class templates (the weight vectors $w_k$ of the last layer), and project the penultimate layer activations onto that plane. This reveals the clustering geometry directly.

The logit for class $k$ is $x^T w_k$, which can be related to the squared Euclidean distance between the activation vector $x$ and the template $w_k$:

A model is well-calibrated if, among all predictions it makes with 80% confidence, approximately 80% are actually correct. Modern neural networks are notoriously over-confident: they routinely predict 99% confidence on examples they get wrong.

Guo et al. (2017) showed that post-hoc temperature scaling — dividing logits by a learned temperature $T$ before the softmax — can fix calibration. But this requires a held-out validation set and a separate calibration step.

Müller et al. discovered that label smoothing provides a similar calibration benefit for free, as a side effect of training. The expected calibration error (ECE) drops substantially without any post-processing. On CIFAR-100 with ResNet-56, label smoothing with $\alpha = 0.05$ reduces ECE from 0.150 to 0.024 — nearly matching temperature scaling at $T = 1.9$ which achieves 0.021.

The calibration benefit has a direct practical payoff in machine translation. Vaswani et al. had reported that label smoothing with $\alpha = 0.1$ improved BLEU scores in their Transformer model despite worse perplexity. This seems paradoxical — how can a model with worse predictions produce better translations?

The answer is calibration. Beam search, the decoding algorithm used in translation, relies on model confidence to rank candidate sequences. An over-confident model assigns disproportionately high probability to its first guess, preventing beam search from exploring better alternatives. A well-calibrated model spreads probability more truthfully, giving beam search better signal to work with.

Knowledge distillation trains a smaller student network to mimic the soft outputs of a larger teacher network. The key idea is that the teacher's soft predictions carry "dark knowledge" — information about which incorrect classes are more similar to each other. A teacher that gives 10% probability to "dog" and 0.1% to "truck" when classifying a cat image is telling the student that cats look somewhat like dogs but nothing like trucks.

Label smoothing destroys exactly this information. By forcing the penultimate layer to place every example equidistant from all incorrect class templates, label smoothing ensures that the teacher's soft predictions treat all wrong classes as equally wrong. The "dark knowledge" about inter-class similarities is erased.

The experiment is striking: a ResNet-56 teacher trained with label smoothing on CIFAR-10 achieves better accuracy than one trained with hard targets. But when this better teacher distills knowledge into an AlexNet student, the student performs worse than one trained with the hard-target teacher. A better teacher produces a worse student — because the teacher's logits have lost the relative information that makes distillation effective.

To go beyond visual intuition, the authors quantified the information erasure by estimating the mutual information between training examples and their logit differences. Mutual information measures how much knowing the input tells you about the output beyond just knowing the class label.

For a network trained with hard targets, the mutual information stays high throughout training. Different training examples from the same class produce different logit patterns — a cat lying in grass has different inter-class similarities than a cat sitting on a chair. This example-specific information is what makes distillation powerful.

For a network trained with label smoothing, the mutual information rises initially (as the network learns to classify) but then drops sharply. Eventually it approaches $\log(2)$ — the information content of a single binary label, meaning the logits carry almost no information beyond "correct class vs everything else." Every cat looks the same in the logit space, regardless of context.

The paper validates its findings across three domains, three architectures, and three datasets. In image classification, AlexNet on CIFAR-10 and ResNet-56 on CIFAR-100 both showed the tight clustering phenomenon with label smoothing. In machine translation, the Transformer on English-German showed both improved BLEU scores and better calibration with label smoothing. On ImageNet with Inception-v4, label smoothing reduced ECE from 0.071 to 0.035.

An especially interesting case is semantically similar classes on ImageNet. When visualizing two poodle breeds (toy poodle and miniature poodle) together with a dissimilar class (tench fish), hard targets produce overlapping clusters for the similar classes with a continuous gradient of similarity toward the dissimilar class. Label smoothing forces even these similar classes into an arc shape, erasing the gradient of similarity. You can no longer measure "how much this poodle resembles a tench" — a piece of information that hard-target networks naturally preserve.