Gaussian Error Linear Units (GELUs)

By 2016, the ReLU activation had become the workhorse of deep learning. Its formula is disarmingly simple: $\text{ReLU}(x) = \max(0, x)$. If the input is positive, pass it through unchanged. If negative, output zero. This simplicity made it fast and effective — far better than the sigmoid functions that came before, which suffered from vanishing gradients in deep networks.

But ReLU has a sharp edge. At $x = 0$, the function makes a sudden, discontinuous jump in its derivative — from 0 to 1. This hard kink means the function is not smooth: its gradient flips instantaneously. For values just below zero, the neuron is completely dead. For values just above, it is fully alive. There is no nuance near the boundary.

Meanwhile, stochastic regularizers like dropout were applied separately. Dropout randomly zeroes neurons during training to prevent overfitting — but it does so blindly, without considering whether a neuron's signal is large or small. The activation decision and the regularization decision were two independent mechanisms, never talking to each other.

The key insight behind GELU is beautifully simple: merge the activation function and the regularizer into one operation.

Start with a thought experiment. Imagine that for each neuron, we flip a biased coin to decide whether to keep or drop the input — but the coin's bias depends on the input itself. Large positive inputs are almost certainly kept. Large negative inputs are almost certainly dropped. Values near zero have roughly a 50/50 chance.

Formally: multiply the input $x$ by a mask $m \sim \text{Bernoulli}(\Phi(x))$, where $\Phi(x) = P(X \leq x)$ is the cumulative distribution function (CDF) of the standard normal distribution. This means: the probability of keeping $x$ equals the probability that a random standard normal variable is less than or equal to $x$.

This is a stochastic regularizer — like dropout, but input-aware. Now take its expected value to get a deterministic function:

$\mathbb{E}[x \cdot m] = x \cdot \Phi(x) + 0 \cdot (1 - \Phi(x)) = x \cdot \Phi(x)$

That is the GELU. No randomness at inference time — just a smooth, deterministic curve that encodes the probabilistic intuition.

Now that we understand the intuition — the probabilistic coin flip that keeps or drops each input based on its value — let us write down the exact formula. The expected value of that stochastic process gives us a clean, deterministic activation function.

Computing the exact error function can be slow in some frameworks, so the paper provides two fast approximations:

Tanh approximation (more accurate): $\text{GELU}(x) \approx 0.5x\left(1 + \tanh\left[\sqrt{2/\pi}\left(x + 0.044715x^3\right)\right]\right)$

Sigmoid approximation (faster): $\text{GELU}(x) \approx x \cdot \sigma(1.702x)$

Today, most frameworks implement the exact version efficiently, but these approximations played a key role in early adoption — especially the tanh form, which was used in BERT's original implementation.

Three properties set GELU apart from ReLU and ELU:

Smoothness. GELU is infinitely differentiable everywhere. There is no kink, no sudden change in gradient. This makes optimization landscapes smoother, helping gradient-based optimizers find better paths. ReLU's hard corner at zero can cause oscillations in training.

Non-monotonicity. GELU has a small dip below zero near $x \approx -0.17$. This means it can output slightly negative values — a counterintuitive property that allows it to encode subtle "anti-features." ReLU and ELU are strictly monotonic: they never decrease as the input increases.

Curvature everywhere. ReLU is perfectly linear for $x > 0$ — there is zero curvature in the positive domain. GELU curves gently even for positive inputs, gradually approaching the identity line $y = x$ only as $x \to \infty$. This curvature at all points may help the network approximate complex functions more easily.

GELU sits in a clear lineage. Understanding where it came from reveals why it works:

ReLU — $\max(0, x) = x \cdot \mathbf{1}_{x > 0}$ — gates by sign. The indicator function $\mathbf{1}_{x > 0}$ is a hard step: exactly 0 or 1. This is actually GELU with $\sigma \to 0$: as the Gaussian CDF sharpens to a step function, $\Phi(x) \to \mathbf{1}_{x > 0}$, and GELU collapses to ReLU. So ReLU is a special case of GELU.

ELU — smooths ReLU's negative side with an exponential curve: $\alpha(e^x - 1)$ for $x < 0$, but remains linear for $x > 0$. It handles the dead-neuron problem but has no probabilistic motivation.

SiLU (Sigmoid Linear Unit) — $x \cdot \sigma(x)$, where $\sigma$ is the logistic sigmoid. Also introduced in this paper, the SiLU uses a logistic CDF instead of a Gaussian CDF. It was later independently proposed as "Swish" by Google Brain. Similar in spirit to GELU but slightly less accurate because the logistic CDF is a looser approximation of the Gaussian.

The paper tested GELU against ReLU and ELU across five domains:

• MNIST classification — 8-layer feedforward network: GELU achieved the lowest training and validation loss both with and without dropout. It also showed better robustness to noised inputs.
• MNIST autoencoding — deep autoencoder (1000→500→250→30→250→500→1000): GELU significantly outperformed both competitors at multiple learning rates.
• Twitter POS tagging — 2-layer network with pretrained word vectors: GELU achieved 12.57% error vs 12.67% for ReLU and 12.91% for ELU.
• TIMIT speech recognition — 5-layer, 2048-neuron classifier: GELU achieved 29.3% error vs 29.5% ReLU and 29.6% ELU.
• CIFAR-10/100 — convolutional networks: GELU achieved 7.89% error on CIFAR-10 vs 8.16% ReLU; on CIFAR-100 with wide residual networks, GELU achieved 20.74% vs 21.77% ReLU.

The consistent pattern: GELU matched or outperformed on every single benchmark.

Where did GELU end up? Inside the Transformer.

The original Transformer used ReLU in its feed-forward network (FFN) layers. When BERT was published in 2018, it swapped ReLU for GELU in the FFN — and achieved state-of-the-art results on 11 NLP benchmarks. GPT-2 followed suit. By GPT-3, GELU was the unchallenged default.

Why was GELU such a good fit for Transformers? The feed-forward network in each Transformer block is where the model stores and retrieves factual knowledge. GELU's smooth, value-weighted gating gives the FFN more expressive power than a hard ReLU gate. The smoothness also helps with the very deep stacks (12–96 layers) that Transformers use, because smooth gradients flow more reliably through many layers of residual connections.

Today, GELU is to Transformers what ReLU was to convolutional networks: the activation you use unless you have a specific reason not to.