Rectified Linear Units Improve Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) are the building blocks of deep belief networks. Each RBM has a visible layer (the input) and a hidden layer (the learned features), with symmetric connections between them but no connections within the same layer. Training uses Contrastive Divergence: the model reconstructs the input from the hidden features and adjusts weights to minimize the reconstruction error.

The catch: each hidden unit was binary — it could only say "yes, I detect this feature" (1) or "no" (0). If one image of a face is twice as bright as another, both activate the same hidden unit the same way. The intensity information is destroyed. This is fine for simple binary patterns, but catastrophic for real-world images where brightness, contrast, and relative intensities carry meaning.

The key insight comes in three steps, like building a staircase:

Step 1 — Binomial units: Take a binary unit and make N copies sharing the same weights and bias. The total activation is the count of "on" copies — an integer between 0 and N. More copies means more information, but N must be fixed in advance.

Step 2 — Stepped Sigmoid Units: Now take infinitely many copies, but give each copy a progressively more negative bias offset: −0.5, −1.5, −2.5, … The total activity of all copies has a closed form that looks like a smoothed staircase. The learning rules stay identical to binary RBMs — this is mathematically elegant because the gradient decomposes into independent per-copy gradients.

Step 3 — Noisy ReLU (NReLU): Computing infinite sigmoid copies is expensive. But the staircase is well approximated by a simple function: $\max(0,\; x + \mathcal{N}(0, \sigma^2))$ — a rectified linear unit with Gaussian noise. The noise makes sampling easy; the rectification (clipping at zero) creates natural sparsity.

Strip away the noise and the RBM context, and the core idea is beautifully simple. Given a total input $x$ to a neuron (the weighted sum of inputs plus bias), ReLU computes:

Compare this with the sigmoid function $\sigma(x) = 1 / (1 + e^{-x})$, which squashes everything into the range (0, 1). With sigmoid, a neuron receiving an input of 10 gives an output of 0.99995 — almost identical to an input of 5 (output 0.9933). ReLU outputs 10 and 5 respectively, preserving the ratio.

This matters enormously when you stack layers. Binary or sigmoid representations progressively destroy how strongly each feature fires, keeping only a compressed echo. ReLU representations preserve exact magnitudes, so a feature that fires twice as hard in one image stays twice as hard in every subsequent layer.

One of the paper's most elegant observations is intensity equivariance. If you scale all pixel intensities in an image by a factor $\alpha > 0$, then every neuron's input also scales by $\alpha$. For a zero-bias ReLU unit:

• If the unit was "off" (input < 0), scaling by $\alpha$ keeps it off.
• If the unit was "on" (input > 0), its output scales by exactly $\alpha$.

So the entire activation pattern scales uniformly — the network's internal representation varies in the same way as the input intensity. This is intensity equivariance: the representation is a faithful, proportional map of the input.

Binary units lack this property entirely. Whether a pixel's intensity is 0.5 or 5.0, a binary unit either fires or doesn't — the ratio is lost. For tasks like face verification, where the same face might appear under very different lighting, intensity equivariance means ReLU features can be compared using the cosine of the angle between activation vectors, which is inherently intensity-invariant.

When a ReLU unit receives a negative total input, its output is exactly zero — not "close to zero" like a sigmoid, but exactly zero. In a typical trained network, around 50% of ReLU units are off for any given input. This property is called sparsity.

Why is sparsity valuable? Think of it as a room full of experts. When a sigmoid network processes an image, every expert mumbles an opinion (even the irrelevant ones). When a ReLU network processes the same image, half the experts stay silent — only the relevant ones speak. The result is a cleaner, more interpretable, and more efficient representation.

Sparsity also helps with computation: multiplying by zero is free. And it introduces a form of natural regularization — the network can't rely on all features simultaneously, which reduces overfitting.

ReLU's gradient is wonderfully simple:

• For $x > 0$: the gradient is exactly 1. The signal passes backward through the network without shrinking or growing. No vanishing gradient.
• For $x \leq 0$: the gradient is 0. The unit is off and ignores the update.

Contrast this with sigmoid, whose maximum gradient is 0.25 (at x = 0) and shrinks exponentially as |x| grows. When you chain many sigmoid layers, each one multiplies the gradient by at most 0.25 — after 10 layers, the gradient is smaller by a factor of $0.25^{10} \approx 10^{-6}$. This is the vanishing gradient problem.

ReLU doesn't multiply — it either passes the gradient unchanged (1) or blocks it (0). There is no "in between" that slowly strangles the learning signal. This is why ReLU networks can be trained with simple backpropagation at depths that would cause sigmoid networks to stall.

An RBM defines an energy function over visible units $\mathbf{v}$ and hidden units $\mathbf{h}$. Lower energy means higher probability — the model learns by pushing the energy of training examples down and the energy of everything else up.

For binary RBMs, Contrastive Divergence samples binary hidden states given the input, reconstructs the visible layer, then re-samples the hidden layer. The weight update is the difference between two correlations: the "positive" phase (data-driven) and the "negative" phase (reconstruction-driven).

The beautiful thing is that this learning rule works unchanged for NReLU units. The Stepped Sigmoid derivation guarantees it: since each NReLU is a limit of infinitely many tied binary units, the gradient decomposes identically. You simply replace the binary sampling step with: sample $h_j = \max(0, \mu_j + \mathcal{N}(0, \sigma_j^2))$ where $\mu_j$ is the unit's mean total input. The rest of the algorithm stays the same.

Nair & Hinton tested NReLU on two challenging vision benchmarks:

Jittered-Cluttered NORB — synthetic 3D object recognition with 5 classes of toys photographed in stereo under varying viewpoint, lighting, and clutter. The objects were randomly jittered in position, scale, and brightness. NReLU consistently outperformed binary units: 5.2% lower error without pre-training, 2.2% lower error with pre-training. Remarkably, NReLU without pre-training was better than binary units with pre-training.

Labeled Faces in the Wild (LFW) — real-world face verification: given two face photos, predict "same person" or "different person." Faces vary wildly in lighting, pose, expression, and background. NReLU models showed better accuracy, and crucially used cosine similarity between feature vectors — directly exploiting intensity equivariance to handle lighting variation.

Every major neural network architecture today uses a rectified activation function — whether standard ReLU, GELU, SiLU, or a variant. The simple idea of "clip negatives to zero" is now as fundamental to deep learning as backpropagation itself. What Nair and Hinton showed was not just a better activation function, but a design principle: simplicity, sparsity, and preserving signal magnitude matter more than biological plausibility.