Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

A vanilla RNN updates its hidden state by completely overwriting it at each step:

$h_t = \tanh(W x_t + U h_{t-1})$

This means the network replaces its entire memory every time step. Information from early in the sequence must survive through a chain of matrix multiplications and squashing functions. After enough steps, gradients either vanish (the signal decays to nothing) or explode (it grows uncontrollably). In practice, vanilla RNNs struggle to connect events more than 10–20 steps apart.

By 2014, two gated alternatives had emerged: the well-established LSTM (1997) and the brand-new GRU (2014). Both add gates — learned switches that control information flow — but they differ in how many gates they use and what each gate controls.

The LSTM unit maintains a separate memory cell $c_t$ alongside the hidden state $h_t$. Four components control it:

• Forget gate $f_t$: decides how much of the old memory to keep. Think of a shredder — turn it up and old information is destroyed; turn it down and everything is preserved.
• Input gate $i_t$: decides how much of the new candidate content $\tilde{c}_t$ to write into the cell. Like a mail slot — it controls what gets in.
• Cell update: $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$. Old memory (partially forgotten) plus new content (partially admitted).
• Output gate $o_t$: decides how much of the cell's content to reveal as the hidden state. A tinted window — the filing cabinet is there, but you only see what the gate lets through: $h_t = o_t \odot \tanh(c_t)$.

This design is powerful: the cell can carry information across hundreds of steps unchanged. But it requires three gating vectors ($f_t$, $i_t$, $o_t$) plus a candidate ($\tilde{c}_t$), meaning four weight matrix multiplications per step — a heavy computational footprint.

The GRU asks: can we get the benefits of gating without a separate memory cell and without an output gate? Its answer is yes, using just two gates:

Update gate $z_t$ — controls the blend between old and new. When $z_t \approx 1$, the unit adopts the new candidate entirely; when $z_t \approx 0$, it copies the old state unchanged. This single gate plays both the roles of the LSTM's forget and input gates — they are tied together: whatever fraction you forget is exactly the fraction you fill with new content.

Reset gate $r_t$ — controls how much of the previous state is visible when computing the candidate. When $r_t \approx 0$, the unit behaves as if it is reading the very first element of the sequence — it can make a fresh start. When $r_t \approx 1$, all previous context flows in.

Think of a highway with a single adjustable lane divider (update gate): one side carries old information forward, the other carries new information in. The reset gate is a filter on the rearview mirror — it controls how much of the road behind you influences your next turn.

The two units share a fundamental idea: additive updates create shortcut paths that let gradients flow across many time steps without vanishing. In both units, the new state is not computed by overwriting; it is blended with the old state via learned gates.

But they differ in architecture:

• LSTM has a separate memory cell $c_t$ that is never directly exposed; the output gate controls what the rest of the network sees. GRU has no separate cell — the hidden state is the memory.
• LSTM's forget and input gates are independent: you can forget 80% and write 50%. In GRU they are tied: forget 80% means write exactly 80%.
• LSTM uses 4 weight matrices per gate/candidate (≈ 4 matrix multiplies per step). GRU uses 3 (update gate, reset gate, candidate). Fewer parameters for the same hidden size.

The paper highlights that the most important feature shared by LSTM and GRU — and absent from vanilla RNNs — is their additive update path.

In a vanilla RNN, $h_t$ is computed by a nonlinear function of $h_{t-1}$, creating a chain of bounded nonlinearities. Back-propagating through this chain multiplies many Jacobians together, and if each has spectral radius < 1, the gradient shrinks exponentially — the vanishing gradient problem.

Both LSTM and GRU replace this chain with an additive shortcut: part of the old state is added directly to the new state. When the gate is saturated near 1, the gradient flows through unchanged — like a highway that bypasses city traffic. This is the same principle behind residual connections in deep feedforward networks, applied here to the time axis.

To make the comparison fair, the authors gave all three models approximately the same total number of parameters — about 20K for music tasks and 169K for speech tasks. Because GRU has fewer matrices per unit than LSTM, it can afford more hidden units at the same parameter count: 46 GRU units vs 36 LSTM units for music, 227 vs 195 for speech.

They tested on two domains:

• Polyphonic music modeling (4 datasets: Nottingham, JSB Chorales, MuseData, Piano-midi): each time step is a multi-hot binary vector representing which notes are being played simultaneously.
• Speech signal modeling (2 Ubisoft datasets): raw one-dimensional audio. The model reads 20 consecutive samples and predicts the next 10. One dataset has sequences of length 500 (short), the other 8,000 (long) — a direct test of long-range dependency capture.

All models were trained with RMSProp, weight noise (σ = 0.075), and gradient clipping (norm ≤ 1). Learning rates were tuned by random search.

The results tell a clear two-level story:

Level 1 — Gates matter. On the challenging speech datasets, both LSTM and GRU dramatically outperformed the vanilla tanh unit. On Ubisoft B (long sequences), tanh achieved a test negative log-likelihood of 7.62 while GRU reached 0.88 and LSTM 1.26 — an order of magnitude better. The gating mechanism is not optional; it is essential for non-trivial sequence modeling.

Level 2 — GRU ≈ LSTM. Across all six datasets, neither gated unit consistently dominated the other. GRU won on most music datasets and on the long speech dataset; LSTM won on the short speech dataset. On music, the differences were small. The authors' conclusion: the choice between them depends on the task and dataset, but GRU's simpler design carries no systematic penalty.

The learning curves revealed an additional advantage: GRU often converged faster in both wall-clock time and number of updates, likely because fewer parameters per unit means each gradient step is cheaper and the optimization landscape is smoother.

Before this paper, GRU had only been tested in one context (machine translation by Cho et al., 2014). This systematic evaluation across music and speech gave the community the empirical confidence to adopt GRU broadly. The practical implications were significant:

• Faster training: fewer parameters per unit means faster gradient computation and often faster convergence.
• Smaller models: for the same hidden dimension, GRU uses ~25% fewer parameters than LSTM — crucial for edge deployment.
• Design philosophy: the paper showed that architectural complexity does not automatically buy better performance. Sometimes simpler is just as good — a lesson that echoes through modern architecture design.