On the Difficulty of Training Recurrent Neural Networks

A recurrent neural network processes sequences by maintaining a hidden state that gets updated at every time step. To learn, we unroll the network across time and backpropagate — a process called Backpropagation Through Time (BPTT). The gradient of the loss at time $t$ with respect to parameters at an earlier step $k$ involves a product of Jacobian matrices, one for each step between $k$ and $t$:

Think of it like compound interest. If each Jacobian factor amplifies the signal by a factor $\lambda > 1$, after $t - k$ steps the signal grows as $\lambda^{t-k}$ — exponential explosion. If $\lambda < 1$, it shrinks as $\lambda^{t-k}$ — exponential decay. A product of 100 matrices with $\lambda = 1.1$ amplifies by $1.1^{100} \approx 13{,}781$. With $\lambda = 0.9$ it shrinks to $0.9^{100} \approx 0.000027$.

More precisely, if $\lambda_1$ is the largest eigenvalue of $\mathbf{W}_{\text{rec}}$ and $\gamma$ is the upper bound on $|\sigma'(x)|$ (for tanh, $\gamma = 1$; for sigmoid, $\gamma = \tfrac{1}{4}$), then:

• Sufficient condition for vanishing: $\lambda_1 < \frac{1}{\gamma}$ — long-term components decay exponentially.
• Necessary condition for exploding: $\lambda_1 > \frac{1}{\gamma}$ — without this, gradients cannot explode.

The paper reveals a striking geometric pattern. When gradients explode, they do so along a specific direction $\mathbf{v}$ — the eigenvector of the largest eigenvalue. The curvature of the error surface explodes along the same direction, creating a steep wall in the loss landscape.

Imagine a wide, gentle valley with a single sheer cliff face cutting across it. Standard SGD computes a step proportional to the gradient. Near the cliff, that gradient is enormous, and the step launches you clear across the valley — or worse, out of it entirely. Training becomes unstable: the parameters bounce wildly instead of converging.

The key insight is that the valley is wide on both sides of the wall. If instead of taking the enormous step, you cap the step size and take a small step in the gradient direction, you land back in the smooth region beside the wall, where SGD can resume normal exploration.

The authors borrow tools from dynamical systems theory to explain when gradient explosions happen. An RNN's hidden state behaves like a dynamical system: given a set of parameters, the state converges toward attractors — stable equilibria. The parameter space is divided into basins of attraction, each pulling the state toward a different attractor.

As parameters change during training, the landscape of attractors shifts smoothly — almost everywhere. At special points called bifurcation boundaries, attractors appear, disappear, or change shape abruptly. When the training path crosses a boundary between two basins of attraction, a tiny parameter change causes a huge change in the hidden state, which means the gradient explodes.

Think of it like a marble rolling on a landscape with two bowls. Most of the time, tilting the landscape a little just shifts where the marble settles. But at the ridge between the bowls, an infinitesimal tilt sends the marble to an entirely different bowl — a discontinuous jump. That ridge is the bifurcation boundary, and the jump is the gradient explosion.

The geometric insight — a wide valley with a single steep wall — suggests an elegant fix. When the gradient norm spikes (indicating you've hit the wall), don't take the enormous step. Instead, rescale the gradient so its norm equals a fixed threshold, then step. You preserve the direction (still descending) but limit the magnitude (staying near the wall instead of flying across the valley).

The algorithm is strikingly simple:

Why does something so simple work? Because of the valley geometry. The wall is narrow — the explosion happens in a localized region of parameter space. On both sides of the wall, the error surface is smooth and well-behaved. A clipped step from the wall just pushes you back to the smooth side, where standard SGD works fine.

Compared to second-order methods like Hessian-Free optimization, clipping has three advantages: it is trivially cheap to compute, it handles sudden curvature changes (which smoothed Hessian estimates miss), and it works even when the gradient and curvature grow at different rates (which would cause a second-order correction to itself explode).

Clipping tames explosions, but what about the opposite problem — vanishing gradients? When gradient components shrink exponentially, the model cannot learn dependencies between distant time steps.

The paper introduces a regularization term $\Omega$ that penalizes the Jacobian whenever it fails to preserve the gradient norm as the error signal travels backward through time. The idea: force the back-propagated error signal to neither grow nor shrink as it passes through each time step.

Two design choices make this regularizer practical. First, it only penalizes the Jacobian's behavior in the direction of the current error gradient, not across all directions. This avoids the expensive requirement of making all eigenvalues close to 1. Second, the paper uses only the "immediate" partial derivative for efficiency — treating the hidden state and error as constants when differentiating $\Omega$ with respect to $\mathbf{W}_{\text{rec}}$.

There is a natural tension between the two solutions. Preventing vanishing gradients means keeping the model sensitive to distant inputs — but this also means operating near basin boundaries where explosions are more likely. The paper combines both: the regularizer pushes the model toward richer regimes where long-term memory is possible, and clipping acts as a safety net when the inevitable explosions occur.

The combined approach — SGD-CR (SGD + Clipping + Regularization) — was tested on both synthetic pathological benchmarks and real-world tasks.

On the temporal order problem, which requires remembering symbols across long sequences, SGD alone fails beyond length 20. SGD with clipping (SGD-C) improves but still struggles. SGD-CR achieves 100% success on sequences up to 200 steps — and generalizes to sequences twice as long (400 steps) without retraining.

On real-world tasks, clipping alone improved performance on polyphonic music prediction (Piano-midi.de, Nottingham, MuseData) and character-level language modeling (Penn Treebank). Adding the regularizer brought further gains, especially on the modified task of predicting the 5th-next character, where long-range dependencies are more important.

The vanishing gradient problem was first identified by Bengio et al. (1994), which showed that gradient-based learning in RNNs becomes increasingly difficult as the time gap between relevant events grows. That paper provided the theoretical foundation — eigenvalue conditions for gradient decay — that Pascanu et al. extend to the exploding case.

LSTM (1997) solved the vanishing gradient architecturally by introducing gated memory cells with a constant error carousel. However, LSTM did not address exploding gradients explicitly. Hessian-Free optimization (Martens and Sutskever, 2011) handled both problems through second-order information but was far more expensive.

Gradient clipping bridges the gap: it is cheap enough for standard SGD, handles explosions that architectural changes like LSTM do not address, and complements rather than replaces these other approaches.

Gradient clipping outlived the RNN era it was designed for. When Transformers replaced RNNs as the dominant architecture, clipping came along — not because Transformers have the same Jacobian chain problem, but because deep networks of any kind can encounter sudden gradient spikes during training due to unusual mini-batches, learning rate schedules, or loss landscape geometry.

Today, torch.nn.utils.clip_grad_norm_ is called in virtually every large-scale training loop. GPT, Claude, Gemini, LLaMA — all use gradient clipping. It is one of those rare ideas that is so simple and so effective that it became invisible infrastructure, the kind of thing every practitioner uses without thinking about the paper that introduced it.