Recurrent Neural Networks (RNNs): A Gentle Introduction and Overview

A feedforward network sees its whole input at once and forgets it as soon as it answers. That is fine for a photo, but useless for a sequence: a sentence, a genome, a stock price, handwriting. There, the meaning of each element depends on what came before it, and sequences do not come in one fixed length.

A recurrent neural network solves this with sequential processing. It reads one element per time step and keeps a hidden state $\mathbf{H}_t$, a vector that summarises everything seen so far. At each step the new state is computed from two things: the current input $\mathbf{X}_t$ and the previous state $\mathbf{H}_{t-1}$. Information no longer flows only forward; it loops back.

The crucial detail is that the same weights are used at every step. This is weight sharing, and it is what lets one network handle a sequence of any length.

Drawing the loop out, one copy per time step, is called unrolling. It adds no new machinery; it is simply the same cell drawn $T$ times, with each copy handing its state to the next. Unrolled, an RNN looks like a very deep feedforward network whose layers all share their weights, and that picture is the key to training it.

Weight sharing is also a statement about the world. By using one set of weights everywhere, the network assumes that a pattern means the same thing whether it appears at step 3 or step 300. The price is that whatever links an early input to a late output, the temporal dependency, has to be carried through every copy in between. How well that link survives the trip is the subject of the next two sections.

Once the network is unrolled, ordinary backpropagation almost works. The paper calls the result Backpropagation Through Time (BPTT). The loss is summed over the time steps, $\mathcal{L}(\mathbf{O},\mathbf{Y}) = \sum_{t=1}^{T}\ell_t(\mathbf{O}_t,\mathbf{Y}_t)$ (Eq. 5). The gradient for the output weights $\mathbf{W}_{ho}$ is easy, because each output depends only on its own step (Eq. 6).

The recurrent weights are harder. A change to $\mathbf{W}_{hh}$ affects $\mathbf{H}_1$, which affects $\mathbf{H}_2$, and so on, so the loss at step $t$ depends on $\mathbf{W}_{hh}$ through every earlier step $k$. The chain rule then produces a sum over all those paths, and each path carries the factor $\partial \mathbf{H}_t / \partial \mathbf{H}_k$: how much the state at step $k$ still moves the state at step $t$.

The paper then simplifies this product to a matrix power, $\left(\mathbf{W}_{hh}^{\top}\right)^{t-k}$ (Eq. 11–12). Keep this form in mind, because it quietly drops something: the derivative of the activation $\phi_h$ at every step. It is the linear picture of the recurrence, and the interactive in the next section starts from exactly this form before adding the activation back.

Computing all these paths for a long sequence is expensive, and powers of $\mathbf{W}_{hh}$ are numerically unstable. The practical answer the paper describes is Truncated BPTT: put an upper bound on how many steps the gradient may flow back. Think of it as a moving window over the past. Anything before the cut-off step is simply not taken into account, which is the same as limiting the number of layers in the unrolled network.

The paper names the central problem of RNNs plainly. The factor $\partial \mathbf{H}_t / \partial \mathbf{H}_k$ means multiplying matrices over a potentially very long sequence. If the values involved are smaller than 1, the gradient shrinks at every step and eventually disappears: the vanishing gradient. States from far in the past then stop contributing to learning. If the values are larger than 1, the product grows instead, the exploding gradient, and a single update can throw the weights far off.

Vanishing is the quieter and more damaging failure. Training does not crash; the network simply never learns a long-term dependency, because the signal that should teach it never arrives. This difficulty had been analysed years before the survey, most famously in Bengio, Simard and Frasconi's 1994 paper on why gradient descent struggles with long-range dependencies.

One more detail matters in practice. With a tanh activation, every step also multiplies by its derivative, which is at most 1 and close to 0 when the unit is saturated. The paper's Eq. 11 leaves that factor out. Put it back, and the picture changes.

In the linear case the arithmetic is stark: $0.9^{50} \approx 0.005$ while $1.1^{50} \approx 117$. A weight that looks harmless decides whether step 1 has any say at all. With tanh the surprise is that a weight above 1 does not guarantee survival. The larger the weight, the harder the unit saturates, the smaller tanh′ becomes, and the faster the product dies. That is closer to how real RNNs behave: vanishing is the default, not the exception. (In a single unit, tanh cannot keep exploding; in real networks with many units it still can, which is why remedies for explosion exist.)

Two remedies are shown in the interactive. The first is gradient clipping, which the survey does not discuss: if the gradient's norm exceeds a threshold, rescale it down. It keeps an exploding update from wrecking the weights, but it scales every step by the same factor, so it cannot bring back a vanished signal. The second is the Truncated BPTT window the paper describes. It keeps training stable, but only by declaring the distant past irrelevant.

The Long Short-Term Memory cell (LSTM) changes the product itself. Next to the hidden state it keeps a second vector, the cell state $\mathbf{C}_t$, and it controls what flows in and out of it with a gating mechanism. A gate is a vector of numbers between 0 and 1, produced by a sigmoid from the current input and the previous hidden state, and multiplied element-wise into another vector. Near 0 it blocks; near 1 it lets through.

There are three. The forget gate $\mathbf{F}_t$ decides how much of the old cell to keep (Eq. 15). The input gate $\mathbf{I}_t$ decides how much of a new candidate $\tilde{\mathbf{C}}_t$, built with tanh, to write in (Eq. 14, 16). The output gate $\mathbf{O}_t$ decides how much of the cell to expose as the new hidden state (Eq. 13, 18). According to the survey, because LSTMs keep the error more constant, they let RNNs learn over far more time steps, "way over 1000".

The race shows why the forget gate matters so much. Along the cell, the gradient from step $T$ back to step 1 is simply the product of the forget gates in between. A value of 0.9 looks generous, but $0.9^{99} \approx 3\times10^{-5}$, so the bit is erased; 0.99 leaves about a third of it after a hundred steps. The network can learn to hold a gate near 1 for exactly as long as a piece of information is needed.

The extreme case has a name, although the survey does not use it. In the original 1997 LSTM there was no forget gate at all: the cell fed into itself with a fixed weight of exactly 1, a design Hochreiter and Schmidhuber called the Constant Error Carousel. The error signal could circulate unchanged for as long as the gates stayed shut. The forget gate, added by Gers and colleagues in 2000, is what lets the cell clear itself when a memory is no longer needed.

The survey mentions the GRU only once, as an alternative to LSTM inside seq2seq models. For completeness: the Gated Recurrent Unit of Cho et al. (2014) merges the cell and the hidden state and uses two gates. An update gate $z$ blends the old state with a new candidate, playing the forget and input roles with one knob, and a reset gate decides how much of the old state the candidate may look at. In the race, $1-z$ plays the part of the forget gate.

Both cells reach the same goal by different routes. What they share is the idea that separates them from the vanilla RNN: the path from one memory to the next is controlled by a learned number close to 1, instead of being forced through a weight matrix and a squashing function at every step.

Two further extensions change the shape of the network rather than the cell. A deep RNN stacks recurrent layers. The first layer reads the input, $\mathbf{H}_t^{(1)} = \phi_1(\mathbf{X}_t, \mathbf{H}_{t-1}^{(1)})$; each higher layer reads the layer below at the same step together with its own previous state, $\mathbf{H}_t^{(\ell)} = \phi_\ell(\mathbf{H}_t^{(\ell-1)}, \mathbf{H}_{t-1}^{(\ell)})$; and the output comes from the top layer (Eq. 19–21). Depth now runs in two directions: up through the layers and along time.

A bidirectional RNN adds a second recurrent layer that reads the sequence backwards, from the last element to the first. At every step there are then two states: a forward state $\overrightarrow{\mathbf{H}}_t$ that has seen the past, and a backward state $\overleftarrow{\mathbf{H}}_t$ that has seen the future (Eq. 22–23). The output uses both, concatenated (Eq. 24). This is useful whenever the answer at one position depends on what comes after it, as in filling a gap in a sequence, the example the paper gives.

Many tasks map one sequence to another of a different length: translation, speech recognition, captioning a video. The paper's answer is the encoder–decoder architecture, known as sequence to sequence (seq2seq). An encoder RNN reads the input one element at a time. Its last hidden state, which the paper calls the encoder vector or context, becomes the initial state of a decoder RNN, which then produces the output one element at a time. The paper notes that these RNNs can be LSTMs or GRUs, and lists Google Translate, voice-enabled devices and video labelling among the applications.

The weak point is that single vector. Whatever the length of the input, everything the decoder will ever know about it has to fit into one fixed-length context vector. The paper names this directly as a bottleneck for long sequences. A short sentence fits; a long one gets compressed until details at the start are lost, which is the analogy's index card all over again.

Attention removes the bottleneck by letting the decoder look back at every encoder state, not just the last one. The encoder, typically bidirectional, produces one state per input position, $\mathbf{H}_{t'}$, the concatenation of the forward and backward states (Eq. 25). At each output step $t$, the decoder computes a fresh context vector $\mathbf{C}_t$ as a weighted average of all of them, and uses it together with its previous state and previous output to form its new state (Eq. 26).

The weights $\alpha_{t,t'}$ say how much output step $t$ should attend to input position $t'$. They come from a score between the decoder's previous state and each encoder state, turned into a probability distribution by a softmax. The paper's running example is the alignment between an English sentence and its French translation.

The alignment matrix is one of the most readable things a sequence model produces. Each row shows where the model looked while writing one output word. Most rows sit near the diagonal, but where the two languages order words differently the attention bends to follow the meaning, not the position.

The survey collects six ways to compute the score. Content-based attention uses the cosine similarity of the two vectors. Additive attention, from Bahdanau et al., passes both through a small network, $\mathbf{v}_a^{\top}\tanh(\mathbf{W}_a[\mathbf{S}_t;\mathbf{H}_{t'}])$. Location-based attention scores from the decoder state alone. The general form inserts a learned matrix between the two, $\mathbf{S}_t^{\top}\mathbf{W}_a\mathbf{H}_{t'}$, the dot product drops that matrix, and the scaled dot product divides the dot product by a constant. The last one is what the Transformer uses, and the reason for the division is worth its own interactive below.

A seq2seq decoder picks each output from a fixed dictionary. For some problems the right answer is not a word at all but a position in the input: which points form the outline of a shape, in what order to visit a set of cities. If the dictionary is fixed, the number of possible positions is fixed too, and a model trained on ten points has no way to name the eleventh.

A Pointer Network (Ptr-Net) reuses the attention weights as the output itself. Instead of mixing the encoder states into a context vector, it takes the softmax over the scores and reads it as a probability of pointing at each input position. The paper writes it with a simplified additive score (Eq. 29). Because the distribution is over the inputs, the size of the "dictionary" grows and shrinks with the input. The survey reports that Pointer Networks were used to compute planar convex hulls, Delaunay triangulations and solutions to the symmetric planar Travelling Salesman Problem.

Attention solved the bottleneck, but the encoder and decoder were still RNNs, and an RNN must process step $t-1$ before step $t$. The Transformer removes the recurrence and keeps only attention. Each position looks directly at every other position of the same sequence through self-attention, so a whole sequence can be processed in parallel, and the path between any two positions is a single step instead of a long chain.

The survey describes the original architecture: a stack of six encoders and six decoders. Each encoder has a self-attention sublayer and a feed-forward sublayer; each decoder adds an encoder–decoder attention sublayer between them. Multi-headed attention runs several attentions in parallel so that each can specialise in a different subspace. Skip connections and layer normalisation keep the deep stack trainable, a final softmax over a vocabulary-sized vector of logits produces the output word, and because nothing is recurrent any more, positional encodings built from sine and cosine waves of several frequencies tell the model where each word sits.

The Transformer's attention is the scaled dot product. The paper states the motivation: when the input to the softmax is large, the softmax can have extremely small gradients, which is a problem for learning. Dot products of long vectors are large. If the entries of a query and a key are random with unit variance, their dot product has a spread that grows like the square root of the dimension $d_k$. Dividing by $\sqrt{d_k}$ puts the scores back on a fixed scale.

One wording in the survey is worth correcting. It describes the divisor as the number of characters of the current word, and its table writes it as $\sqrt{n_{source}}$. In the Transformer paper the divisor is $\sqrt{d_k}$, the dimension of the keys, and that is what the interactive uses.

The survey is a map, and it is best read as one. It gives one consistent notation for a line of ideas that were published in many styles, it derives the gradient that causes the trouble, and its appendices draw an LSTM being assembled gate by gate, a seq2seq model with attention step by step, and the waves behind positional encoding. It ends by recommending the original papers for depth, and by pointing to Vinyals et al.'s StarCraft II agent, AlphaStar, which uses LSTMs, Transformers and Pointer Networks together, as a place where all these pieces meet in one system.

What it does not offer is evidence. There are no experiments of its own, and the architectures are never compared on a common task, so it cannot tell you when a GRU beats an LSTM or how much attention helps on a given problem.

Read in order, the architectures form a chain in which each link answers the weakness of the one before. The vanilla RNN, trained with BPTT, could in principle remember but in practice lost the past to a repeated product. The LSTM and the GRU changed that product with gates. Seq2seq used those cells to map one sequence to another, and in doing so created a new bottleneck: the fixed-length vector. Attention removed the bottleneck, Pointer Networks turned attention into the answer, and the Transformer kept attention and discarded the recurrence that had started the story.

The common thread is the one this paper is organised around: how far a signal can travel before it fades. For the wider context of why depth and gradients became the central questions of the field, see the 2015 Nature review of deep learning.