Learning Long-Term Dependencies with Gradient Descent is Difficult

A recurrent neural network processes sequences by maintaining a hidden state $h_t$ that gets updated at every time step. In principle, this hidden state can carry information from the distant past — the network "remembers" by encoding earlier inputs into $h_t$. Training uses backpropagation through time (BPTT): unroll the network across all time steps and propagate the error gradient backward.

The trouble is that this backward propagation requires multiplying many Jacobian matrices together — one for each time step. By 1994, practitioners had noticed that RNNs could learn short-range patterns (5–10 steps) but failed catastrophically on anything longer. The network would either stop learning entirely (the gradient reaching early time steps was essentially zero) or become numerically unstable (the gradient exploded to infinity).

Bengio, Simard, and Frasconi set out to explain why this happens, not just observe that it happens.

The hidden state evolves as $h_t = f(W_h h_{t-1} + W_x x_t + b)$, where $f$ is a nonlinear activation (like sigmoid or tanh) and $W_h$ is the recurrent weight matrix. When we backpropagate from time step $T$ to an earlier step $t$, the chain rule gives us a product of Jacobian matrices:

$\frac{\partial h_T}{\partial h_t} = \prod_{k=t}^{T-1} J_k$

where each $J_k = \text{diag}(f'(z_k)) \cdot W_h^{\top}$ is the Jacobian at step $k$. The key question is: what happens to this product as $T - t$ grows?

Think of it as compound interest, but applied to gradient magnitude. If each Jacobian matrix shrinks the gradient by a factor of 0.9, then after 50 steps the gradient is $0.9^{50} \approx 0.005$ — practically zero. After 100 steps it's $0.9^{100} \approx 0.00003$. The network simply cannot "see" what happened 100 steps ago.

Conversely, if each matrix amplifies by 1.1, after 50 steps the gradient is $1.1^{50} \approx 117$, and after 100 steps it's $1.1^{100} \approx 13{,}781$. The optimization becomes numerically unstable — weights swing wildly.

Bengio formalized this using the spectral norm (largest singular value) of the Jacobian. When $\|J_k\| < 1$ for all $k$, the product $\|\prod J_k\| \leq \prod \|J_k\|$ shrinks exponentially. When $\|J_k\| > 1$, it grows exponentially. The boundary $\|J_k\| = 1$ is unstable — any small perturbation pushes you to one side or the other.

The deepest insight of the paper is not just that gradients vanish — it's that vanishing gradients and reliable memory are two sides of the same coin.

For an RNN to "remember" information, its hidden state must settle into a stable region — a basin of attraction (or hyperbolic attractor) where small perturbations don't knock the state out. Mathematically, this requires the eigenvalues of the Jacobian at the attractor to be less than 1 in absolute value. But this is exactly the condition that makes gradients vanish!

The paper frames it as a trade-off: if you want robust information storage (eigenvalues well below 1), then gradients decay exponentially and the network can't learn to use that memory. If you want strong gradients (eigenvalues near or above 1), the stored information becomes fragile — noise or new inputs can easily overwrite it.

This is like a filing cabinet with a stiff lock: the lock keeps documents safe from being accidentally knocked out (robust storage), but the same stiffness makes it hard to open the cabinet to file or retrieve anything (learning). A loose lock makes filing easy but documents fall out at the slightest bump.

The activation function plays a critical role. The sigmoid function $\sigma(z) = 1/(1+e^{-z})$ has a maximum derivative of 0.25 (at $z = 0$). For tanh, the maximum is 1.0 (also at $z = 0$). This means:

• With sigmoid: each Jacobian factor is bounded by $0.25 \times \|W_h\|$. Even if $\|W_h\| = 4$, the factor is at most 1.0 — and in practice, activations are rarely at $z = 0$, so the actual bound is much lower. Gradients almost always vanish.

• With tanh: the bound is $1.0 \times \|W_h\|$, which gives slightly more room but still suffers the same exponential behavior.

The flat tails of these saturating functions act as gradient killers: once a neuron enters the saturated region ($|z| \gg 0$), the local derivative drops to near zero, and no gradient can pass through.

Bengio et al. designed a diagnostic task to expose the problem cleanly. The "latch task" works like this: early in a sequence, the network sees a binary signal (0 or 1). Then it processes many irrelevant "distractor" time steps. At the end, the network must output the value of that early signal.

With a time lag of 20 steps, standard BPTT succeeded. With a lag of 50 steps, it failed completely — the gradient had decayed so much that the network couldn't learn to connect the early input to the late output. The network behaved as if the early signal never existed.

This clean failure confirmed the theoretical prediction: the problem isn't just about "not enough training time" — it's a mathematical impossibility for vanilla gradient descent to learn these dependencies beyond a certain lag.

Bengio et al. didn't just diagnose the problem — they also proposed solutions, though the most impactful ones came from researchers inspired by this analysis:

In the paper itself, the authors explored: (1) simulated annealing as a non-gradient optimization method that avoids the vanishing gradient entirely; (2) discrete error propagation using sign changes rather than continuous gradients; and (3) time-weighted penalties that give more learning signal to recent time steps. These worked on toy problems but didn't scale.

Inspired by this analysis, other researchers developed the solutions we still use today:

• LSTM (1997): Hochreiter & Schmidhuber introduced gated memory cells with a "constant error carousel" — a path where the gradient flows unchanged (eigenvalue = 1 along the cell state), bypassing the vanishing gradient entirely.

• Gradient clipping (2013): Pascanu et al. showed that while vanishing gradients are structural, exploding gradients can be handled by simply capping the gradient norm. This was a direct extension of Bengio's analysis.

• Xavier/Glorot initialization (2010): Glorot & Bengio (yes, the same Bengio) showed that initializing weights so each layer preserves the variance of activations delays the onset of vanishing/exploding gradients in deep feedforward networks.

A central concept in the paper is the "hyperbolic attractor" — a fixed point or region in the hidden state space where the dynamics naturally pull the state toward. Think of it as a valley in a landscape: a ball placed anywhere on the hillside will roll down to the valley floor and stay there.

When an RNN stores a bit of information (say, "the input was category A"), it does so by settling into a specific attractor basin. The depth of the valley determines robustness: a deep valley means the state won't be knocked out by noise, but also means gradients can't escape the valley to send learning signals backward through time.

The paper's key figure shows this visually: in the basin of attraction, all trajectories converge to the attractor (good for storage), but the Jacobian eigenvalues are less than 1 everywhere in that basin (bad for gradients).