Explaining and Harnessing Adversarial Examples

In 2014, Szegedy et al. made a startling discovery: take any image correctly classified by a deep neural network, add a carefully crafted perturbation so small that the two images look identical to the human eye, and the network confidently predicts a completely wrong class. A panda becomes a gibbon. A stop sign becomes a speed limit.

Three facts made the phenomenon especially puzzling:

• The perturbations are imperceptible — each pixel changes by less than 1/255 of the dynamic range.
• The same adversarial example fools different architectures trained on different data. The trick is not model-specific.
• Standard defenses — dropout, pretraining, model averaging — offered no protection.

The prevailing theory blamed adversarial examples on deep networks being too nonlinear — highly curved decision boundaries with tiny pockets of misclassification. Goodfellow et al. argued the opposite: the real culprit is that modern networks are too linear.

Why linear? Because we designed them that way. ReLU activations are piecewise linear. LSTM gates operate in their linear regime. Maxout units select among linear functions. Even sigmoid networks are tuned to avoid saturation — staying in the near-linear middle. This linearity makes networks easy to optimize with gradient descent, but it also makes them easy to fool.

Here is the core mathematical argument. Consider a weight vector $\mathbf{w}$ and an input $\mathbf{x}$. Now add a perturbation $\boldsymbol{\eta}$ to get the adversarial input $\tilde{\mathbf{x}} = \mathbf{x} + \boldsymbol{\eta}$. The dot product becomes:

$\mathbf{w}^\top \tilde{\mathbf{x}} = \mathbf{w}^\top \mathbf{x} + \mathbf{w}^\top \boldsymbol{\eta}$

The adversarial perturbation changes the activation by $\mathbf{w}^\top \boldsymbol{\eta}$. To maximize this change while keeping each element of $\boldsymbol{\eta}$ tiny (bounded by $\epsilon$), we set $\boldsymbol{\eta} = \epsilon \cdot \text{sign}(\mathbf{w})$. The resulting activation shift is $\epsilon \cdot \|\mathbf{w}\|_1$.

If $\mathbf{w}$ has $n$ dimensions with average element magnitude $m$, the shift is $\epsilon \cdot m \cdot n$. Each element of $\boldsymbol{\eta}$ is at most $\epsilon$ — invisible — but the total effect grows linearly with dimension $n$. In a space with thousands of dimensions, that tiny per-pixel nudge accumulates into a massive output change.

The linear explanation doesn't just explain adversarial examples — it immediately gives us a recipe to generate them. If linearity is the problem, then linearizing the cost function around the current input gives the optimal perturbation direction.

The idea: compute the gradient of the loss with respect to the input, then take the sign of each element. This gives the direction that maximally increases the loss at each pixel while keeping the perturbation within the $\ell_\infty$ ball of radius $\epsilon$.

One forward pass, one backward pass, one element-wise sign — and the adversarial example is ready. Compare this to the previous method by Szegedy et al., which required expensive L-BFGS optimization over many iterations.

Think of FGSM as climbing the steepest hill on the loss landscape, but taking exactly one step of fixed size $\epsilon$ in every dimension. The sign function ensures every pixel pushes the loss upward by the same amount — it's the most efficient use of a limited perturbation budget.

Devastatingly effective. A simple softmax classifier on MNIST, attacked with $\epsilon = 0.25$, jumps from 1.6% error to 99.9% error — nearly every single example is misclassified. A maxout network, a much stronger model, still reaches 89.4% error under the same attack with 97.6% average confidence on incorrect predictions.

On CIFAR-10 with $\epsilon = 0.1$, a convolutional maxout network reaches 87.15% error with 96.6% confidence on wrong labels. The models are not just wrong — they are confidently wrong.

One of the most puzzling facts about adversarial examples is their transferability: an adversarial image crafted to fool Model A often fools Model B too — even if B has a different architecture and was trained on different data. Under the nonlinearity hypothesis, this makes no sense: why would two very different nonlinear functions share the same blind spots?

The linear explanation makes this natural. Different models trained on the same task learn similar weight vectors, because machine learning generalizes — that's its whole point. If Model A and Model B have similar weight directions, then the adversarial perturbation $\text{sign}(\nabla_\mathbf{x} J)$ for Model A will have a large positive dot product with Model B's weights too. The perturbation moves in a broad direction in input space, not a fragile pinpoint, so it transfers readily.

If FGSM can generate adversarial examples cheaply, why not train on them? The idea is simple but powerful: at each training step, generate adversarial versions of the current batch and train on both the clean and adversarial examples. The model learns to resist the strongest perturbation it can find, not just memorize the training set.

The adversarial training objective mixes the clean and adversarial losses:

The results are striking. On MNIST with a larger maxout network (1600 units per layer):

• Clean test error: improved from 0.94% to 0.78% — adversarial training acts as a regularizer, beating dropout alone.
• Adversarial error: dropped from 89.4% to 17.9% — the model is far more robust.
• The learned weights became more localized and interpretable, focusing on semantically meaningful features rather than diffuse noise-like patterns.

Adversarial training can be understood in multiple ways: as playing a minimax game with an adversary, as minimizing an upper bound on the expected cost over noisy inputs, or as a form of active learning where the model requests labels for the hardest examples.

The paper systematically eliminates competing defenses:

• Dropout and standard regularization: don't change the fundamental linear relationship between input perturbation and output. The model is still easy to fool.
• Generative models: an MP-DBM (a generative model with 0.88% MNIST error) reaches 97.5% error on adversarial examples. Being generative is not protective.
• Ensembles: an ensemble of 12 maxout networks reaches 91.1% error on adversarial examples designed for the full ensemble. Averaging doesn't wash out adversarial perturbations.
• Random noise: training with random $\pm\epsilon$ noise yields 86.2% adversarial error vs 17.9% for FGSM adversarial training. Random noise is "easy" — FGSM finds the "hard" examples.

The one model family that is naturally resistant is RBF networks. They achieve only 55.4% error on adversarial examples because they respond with low confidence (1.2%) when fooled. But RBF networks lack the capacity to generalize well, creating a fundamental tension: easy-to-optimize linear models are easy to attack, while attack-resistant nonlinear models are hard to train.

For logistic regression, the FGSM perturbation is exact (not an approximation). The adversarial training objective becomes minimizing:

$\mathbb{E}_{x,y} \left[ \zeta\!\left(y(||\mathbf{w}||_1 - \mathbf{w}^\top \mathbf{x} - b)\right) \right]$

This resembles $L_1$ regularization, but with a crucial difference: the $L_1$ penalty is subtracted from the activation, not added to the cost. When the model makes confident correct predictions, the penalty can saturate and deactivate. $L_1$ weight decay is more pessimistic — it never turns off, even when the model has sufficient margin. In practice, adversarial training with $\epsilon = 0.25$ works well on MNIST, while $L_1$ with coefficient 0.0025 already causes underfitting.

Not all models are equally vulnerable. The paper draws a clear line between models that can learn robustness and those that cannot:

• Linear models (softmax regression) cannot resist adversarial perturbation. They respond confidently in every direction of input space and have no mechanism to "say I don't know."
• RBF networks naturally resist adversarial examples because they respond confidently only near their learned centers. Far from the data, they default to low confidence. But they sacrifice generalization — they can't transfer knowledge to unseen regions.
• Deep networks with hidden layers can, in principle, learn to resist adversarial perturbation — the universal approximator theorem guarantees this. But they need to be explicitly trained to do so. Standard training doesn't ask for robustness, so the model doesn't learn it.

This creates a spectrum: linear models are high-recall but low-precision (they respond to everything but are easily fooled), while RBF networks are high-precision but low-recall (they respond only to familiar inputs but miss novel ones).

This paper is a turning point. Before it, adversarial examples were a curiosity. After it, they became a research program. The linear explanation provided a unifying framework that made the phenomenon understandable and actionable.