Backpropagation Through Time: What It Does and How to Do It

Werbos opens with a claim that was easy to miss in 1990. Backpropagation is not really about neurons. It is a way to compute, exactly and cheaply, the derivatives of one target quantity with respect to every input and parameter of a system built from known, differentiable steps. The target might be a classification error, and the parameters the weights of a network. But the first practical use, in his 1974 thesis, was fitting a dynamic model of nationalism and social communication.

He first sets up the familiar case with a new notation. A feedforward network computes each neuron from every neuron before it: a weighted sum $\mathrm{net}_i$, then a sigmoid $s(z) = 1/(1+e^{-z})$. Layered networks are the special case where most of those weights are fixed at zero. Training minimises the summed squared error over the training set by steepest descent: find the derivative of the error for every weight, then step each weight against it by a learning rate. His advice is to start the weights small and random, and to raise the rate until the error starts to diverge.

None of this was new. What is unusual, he says, is only how the derivatives are computed: exactly, for all weights, in a single pass through the system.

When Werbos presented the idea to the Harvard faculty in 1972, they doubted that such tangled calculations could be trusted. His answer was a theorem: the chain rule for ordered derivatives. It applies to any system whose quantities can be computed one after another in a fixed order, ending with a target.

The key distinction is between two kinds of derivative. An ordinary partial derivative measures the direct effect: change one variable and hold everything else fixed. An ordered derivative measures the total effect: change the variable, then recompute everything that comes after it. The paper's example is $z_2 = 4 z_1$ and $z_3 = 3 z_1 + 5 z_2$. The direct effect of $z_1$ on $z_3$ is 3. The total effect is 23, because $z_1$ also moves $z_2$, which moves $z_3$ again: $3 + 4 \times 5$. Which of the two matters for training? The total effect, since in a real network nothing downstream stays still.

Werbos gives the ordered derivative of the error with respect to a variable a short name: the feedback to that variable, written with an "F_" prefix. The recipe then becomes mechanical. Start with the feedback to the outputs, which is just output minus target. Walk the neurons in reverse order. Each neuron's feedback collects the feedback of every later neuron it feeds, weighted by the connection. Multiply by the sigmoid's slope, $s'(z) = s(z)(1-s(z))$, to get the feedback to its $\mathrm{net}$ input. The gradient for a weight is then that feedback times the activation flowing into the weight. This backward flow of information is where the name comes from. Each neuron's code has a matching dual subroutine that runs the same connections in reverse.

Two practical notes from the paper are worth keeping. First, any ordered derivative can be checked by perturbation: nudge the variable where it is computed and rerun. Werbos insists on this for debugging, and admits he has not run his own pseudocode. Second, the feedback that reaches the inputs is itself useful. It says how sensitive the target is to each input, which is the basis of sensitivity analysis.

Some classifications improve when the network can use what it saw earlier. Speech and submarine detection are the paper's examples, along with recognising moving objects, which requires comparing the scene at $t$ with the scene at $t-1$. Many good recognisers also refine their picture of the world at $t$ starting from their picture at $t-1$. Even a Kalman filter works this way.

Werbos's clearest illustration comes later in the paper. Picture a ball that rolls under a table and out the far side. A memoryless predictor sees only the current input. Once the ball is out of sight, it has nothing left to reason from. A predictor with memory can keep tracking the ball. Werbos also cites Harold Szu's version: a memoryless person chased by a tiger forgets the tiger as soon as they turn to run.

The solution is a network whose neurons also take inputs from earlier time steps. Werbos's version is deliberately general: a time-lagged recurrent network in which any neuron can read any neuron's value from one or two steps back.

Here is the paper's central observation. A recurrent network still computes its values in a fixed order, as long as that order runs over time as well as over neurons. So the ordered-derivative chain rule applies without change. Backpropagation through time follows from that in three steps.

First, run the network forward over the whole sequence, from $t = 1$ to $t = T$, and keep every intermediate value. Second, treat the run as one large feedforward computation graph, with a copy of the network for each time step. This is unrolling. The copies are not independent. They share the same weights, which is parameter sharing. Third, send the error signal backward from the last step to the first. At every step, add that step's contribution to each shared weight's gradient.

This solves credit assignment over time. An input at step $t$ can now be credited, or blamed, for an error at step $t + k$. The chain of feedback between them carries that temporal dependency into the gradient.

The paper includes practical advice drawn from Werbos's own experience. BPTT fits naturally with batch updates. Derivatives only exist after a complete forward and backward pass, so the weights change once per sequence. The learning rate usually has to be much smaller than for ordinary backpropagation. It also helps to start with the memory weights $W'$ fixed at zero, or at one to force memory, and free them gradually.

Not every moment's error has to count. In speech, for example, only the classification at the end of a phoneme may matter. Setting the output feedback to zero at other times handles that, and the paper notes that any other error criterion can be substituted just as easily. When the data come as many separate sequences, such as words or robot trials, string learning runs the full forward and backward pass on one sequence, updates the weights, and moves on. Only one sequence needs to be stored at a time.

Then Werbos turns to local minima. He argues they matter less than rumour suggests when there are many more training patterns than inputs. But he also shows how easily they can appear, with a training set of just three patterns and two weights.

Each of the three patterns is fitted exactly along one straight line in weight space, and the three lines form a triangle. Werbos says the triangle's vertices correspond roughly to local minima. Rebuilding his example shows when that holds. With plain squared error and a sigmoid, the troughs are soft, and they merge into a single valley in the middle. With a sharper criterion, such as the 1.5 power of the error that the paper itself suggests, the troughs become steep, and each vertex traps descent. This is the paper's general warning in miniature: conflicting patterns can create minima, even in tiny problems.

On speed, Werbos is blunt. Steepest descent is inefficient. He cites heuristics that sped up convergence a hundredfold. Quasi-Newton methods only work for about a hundred weights. Polak–Ribière conjugate gradients need batch learning and very careful line searches. Shanno's conjugate-gradient method worked best for him. He also notes that squared error is not sacred. The 1.5 power works just as well with all the equations above. Adding weight decay to a linear network approaches Kohonen's pseudoinverse as the penalty grows, an approach some statisticians defended as forecasting by analogy with past cases.

The same derivatives serve purposes beyond pattern recognition. In system identification, which Werbos calls neuroidentification, the aim is to learn a forecasting model of a dynamic system. At each step the network receives the observations $X(t)$ and the actions $u(t)$ that were under our control, and it is trained to predict $X(t+1)$. Without memory the forecast can only use the present, which brings us back to the ball under the table. BPTT allows a model with a hidden state. Werbos lists the remaining weaknesses plainly. Forecasts may degrade over several steps. The method does not identify where the noise comes from. And it cannot adapt in real time.

The method also reaches well beyond neural networks. Any model written as a sequence of differentiable equations, such as an econometric model, can be adapted by building its dual. The backpropagation can even sit inside a statistics package, invisible to the user. Models whose equations must be solved simultaneously at each time step need more care. In network terms these are "doubly recurrent" networks, which Werbos later called a simultaneous recurrent network. Networks that settle to an equilibrium, like those of Pineda and Almeida, are special cases. Settling networks are a close relative of the Hopfield network.

In neurocontrol the target is no longer an error. It is a measure of performance, utility, summed over time. The paper's examples range from the energy a robot arm spends to the net profits of the gas industry. Two networks are involved. A plant model predicts how the system responds. An action network looks at the state and chooses the control $u(t)$. Only the action network is being trained.

The procedure is BPTT applied to the combined system. Starting from each initial state, run the action network and the model forward to time $T$. Then sweep back from $T$ to 1 through the dual of the utility function, the dual of the model, and the dual of the action network. The sweep yields the derivative of total utility with respect to every action weight. The weights then move up that gradient, not down, because the goal is to maximise. As Werbos puts it, "backpropagation" names how the derivatives are computed. Nothing about the method requires an error or a classification. The approach is the one used by Nguyen and Widrow for their truck backer-upper and by Jordan.

The demo shows why the full backward sweep matters in control. Pressing the accelerator now changes the speed at once, but the position only later. The cost of arriving too fast shows up later still. A gradient that looks only one or two steps ahead cannot see those consequences, so it learns a controller that does nothing, or one that crashes. Backpropagating through the whole trajectory lets the effect of a late braking decision reach the weights that made it.

Werbos chose BPTT for his own natural-gas application because it is fast and exact. But he names its blind spot. It treats the model as deterministic, so it cannot account for noise in the process being controlled. For that, and for learning while the system is running, he points to the adaptive critic family. These methods learn to estimate future utility, in the style of approximate dynamic programming, and need no backward pass through time.

BPTT's costs follow from its structure, and Werbos states them clearly. The method needs the whole forward history before it can compute anything. Memory therefore grows with the length of the sequence, and learning happens only after the sequence ends. Sparse connections reduce the storage, but they cannot remove it. This rules out true real-time learning, where each observation is used once and then discarded.

Exact derivatives can also be computed forward in time. The method keeps, at every step, how each neuron depends on each weight. This is the Williams–Zipser method, known today as real-time recurrent learning. The paper does not use that name. Werbos had already considered such forward perturbation equations in 1982 and rejected them for neural networks, because their cost grows with the square of the network's size. The approximate alternative is to recast memory problems as control problems and use adaptive critics. These are inexact and complex, but they run forward in time. Werbos's recommendation for the time being is to accept string learning and not pay that price.

What the paper settled was the procedure. Every recurrent neural network trained since uses the unroll-store-sweep procedure described here. It is usually truncated to a fixed window, which is the horizon trade-off from the docking demo, applied to data instead of control.

What the paper left open became the next chapter. It gives exact gradients through long sequences, but it says nothing about their size after a hundred steps. Bengio and colleagues showed that they usually shrink toward zero or blow up, which makes long-range dependencies hard to learn. LSTM was designed to fix that by giving the error a path through time that does not shrink. Both still rely on this paper's backward sweep to compute their gradients.