Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

When you stack many layers, the signal that enters the network gets multiplied by weight matrices over and over. If the variance grows at each layer, the signal explodes exponentially. If it shrinks, the signal — and its gradients — vanish to zero.

Glorot and Bengio's Xavier initialization (2010) addressed this by setting weight variance to $1/n$, where $n$ is the number of inputs to a neuron. But their derivation assumed activations are linear around zero — symmetric, passing positive and negative values equally. That assumption was valid for tanh and sigmoid, the standard activations of 2010.

ReLU breaks this assumption: $\text{ReLU}(x) = \max(0, x)$ zeros out the entire negative half. In expectation, this halves the variance of the signal at every layer. Over $L$ layers the signal shrinks by a factor of $(1/2)^L$. For a 30-layer network that is a factor of about one billionth. The network is mathematically alive but practically dead — all gradients round to zero in floating point.

Standard ReLU has a hard rule: positive inputs pass through unchanged, negative inputs become exactly zero. But is zero always the right slope for the negative side? Leaky ReLU tried a fixed small slope (0.01), but that value was chosen by hand.

PReLU makes the negative slope a learnable parameter $a_i$ for each channel $i$:

Think of it this way: ReLU is a one-way door — positive signals walk through, negative signals are turned away. Leaky ReLU is a door that lets negative signals squeeze through a fixed crack. PReLU is a door whose opening adjusts itself during training.

The gradient for updating $a_i$ is simply the sum of $y_i$ values wherever $y_i \le 0$. This adds almost no computation. For a channel-shared variant, a single parameter $a$ is shared across all channels of a layer — even that one extra parameter per layer measurably improves accuracy.

An interesting finding from the paper: early layers learn large $a$ values (around 0.6), meaning they keep much of the negative signal — they preserve information. Deeper layers learn small $a$ values, becoming more like standard ReLU — they become more selective and discriminative.

The goal of any initialization is simple: keep the variance of the signal roughly constant as it flows through the network, so it neither explodes nor vanishes.

Consider one layer. The output before activation is $y_l = W_l x_l + b_l$, where $x_l$ is a vector of $n_l = k^2 c$ inputs ($k$ is the filter size, $c$ is the number of input channels). If the weights are zero-mean and independent:

$\text{Var}[y_l] = n_l \, \text{Var}[w_l] \, E[x_l^2]$

Xavier assumed $E[x_l^2] = \text{Var}[x_l]$, which holds only if $x_l$ has zero mean. But after ReLU, $x_l = \max(0, y_{l-1})$, which is never negative, so its mean is not zero. Specifically, for a symmetric zero-mean input before ReLU, exactly half the values are zeroed, so:

$E[x_l^2] = \tfrac{1}{2}\,\text{Var}[y_{l-1}]$

Substituting this back:

To keep variance constant across layers, we need:

$\tfrac{1}{2}\, n_l\, \text{Var}[w_l] = 1 \quad \Longrightarrow \quad \text{Var}[w_l] = \frac{2}{n_l}$

This means weights should be drawn from a zero-mean Gaussian with standard deviation $\sigma = \sqrt{2 / n_l}$. Compare this with Xavier's $\sigma = \sqrt{1 / n_l}$ — the only difference is that factor of 2, but it makes all the difference for deep networks.

The same reasoning applies to the backward pass (gradient propagation), where the corresponding condition is $\frac{1}{2}\hat{n}_l \text{Var}[w_l] = 1$ with $\hat{n}_l = k^2 d_l$ (using output channels instead of input channels). Either condition alone is sufficient — satisfying one approximately satisfies the other.

When the activation is PReLU with initial slope $a$, the variance factor changes from $\frac{1}{2}$ to $\frac{1}{2}(1 + a^2)$, because the negative side now scales by $a$ instead of being zeroed:

The paper demonstrates this dramatically. Consider VGG model B with 10 convolutional layers, all using 3×3 filters. The correct standard deviations per He initialization are 0.059, 0.042, 0.029, and 0.021 for layers with 64, 128, 256, and 512 filters respectively. VGG used a constant std of 0.01 throughout. The actual gradient propagated from layer 10 to layer 2 is roughly $1/(1.7 \times 10^4)$ of what it should be — practically zero.

The paper trains a 30-layer model with both initializations. He initialization converges; Xavier completely stalls — the error never decreases, and monitoring confirms all gradients are vanishing. This is not a matter of tuning or patience. The signal is mathematically dead.

The paper trained a 14-layer model and examined the learned $a_i$ coefficients at every layer. Two patterns emerged:

The first convolutional layer — which extracts low-level features like edges and textures — learned large slopes (around 0.6). This means it keeps much of the negative signal. Early layers act as information collectors: they want to preserve as much of the raw input as possible, including patterns that happen to have negative filter responses.

Deeper layers learned progressively smaller slopes, approaching standard ReLU behavior. These layers are more discriminative — they have learned which signals matter and actively suppress the rest. The network self-organizes: preserve early, select late.

This gradient from information preservation to information selection mirrors how humans understand visual hierarchies: first gather all the details, then focus on what matters.

Combining PReLU activations with He initialization, the authors trained models of increasing width and depth. Their best single model achieved 5.71% top-5 error — already better than all prior multi-model ensembles. A six-model ensemble reached 4.94% top-5 error, surpassing the reported human performance of 5.1% on the same dataset.

This was the first time a machine exceeded human accuracy on a large-scale visual recognition challenge. The human baseline was established by a trained annotator using a specialized interface with example images — not a casual guess. The result demonstrated that with the right training fundamentals (proper initialization, proper activations), depth and width could push accuracy to previously impossible levels.

The authors noted that human errors tended to come from fine-grained recognition (e.g., distinguishing 120 dog breeds) and class unawareness, while machine errors came from context-dependent recognition and multi-object scenes.