The History of AI

Reverse-mode automatic differentiation

Era 2 · AI is born (1943–1979) › Neural networks and learning

Lineage
Neurons and networks
Kind
Theory

Who

University of Helsinki

What happened

In a master's thesis at the University of Helsinki, Seppo Linnainmaa presents a method for computing the accumulated effect of rounding errors that is the reverse mode of automatic differentiation. Part of the results appear in English in 1976, in the journal BIT.

Why it mattered

It is the modern, general form of backpropagation: it computes all the derivatives of a composite function in a single backward pass, at a cost of the same order as computing the function itself.

The full story

Linnainmaa was not thinking about neural networks but about a problem in numerical analysis. A program performs thousands of operations on rounded numbers, and each rounding introduces a small error. To know how much each one affects the final result, you need the derivative of the result with respect to every intermediate operation.

The solution was to run the calculation in reverse. The program runs forward once, storing its intermediate values, and then works back from the result to the beginning, applying the chain rule at each operation. That yields the derivatives with respect to every intermediate variable in a single pass, at a cost comparable to the original calculation, instead of repeating the calculation once per variable.

As Linnainmaa told Andreas Griewank, the idea came on a sunny afternoon in 1970 in a park in Copenhagen. Linnainmaa presented the method as a way of estimating rounding errors, not as a cheap way to compute gradients, and the work went almost unnoticed until Griewank got in touch in the late 1980s.

A neural network is a composite function like any other, and training it with backpropagation means applying this reverse mode. That is why Schmidhuber considers Linnainmaa the first to publish what is now called backpropagation.

A chain of computation running left to right, from x to a multiplication by w, a function f, and the result E. Below, dashed arrows run back from E to the left with the derivatives of E with respect to y, a, and w, each obtained from the next.

In reverse mode, values move forward toward the output and derivatives come back from the output to the input.

Under the hood

If the calculation is the chain a = w·x, y = f(a), E = g(y), the chain rule gives ∂E/∂w = g′(y) · f′(a) · x.

Reverse mode walks that chain from the end: it computes ∂E/∂y, uses it to get ∂E/∂a, and uses that to get ∂E/∂w. Each intermediate derivative is computed only once and serves every earlier variable. With many inputs and a single output, like the error of a network with millions of weights, the whole gradient comes at a cost of a few times that of the forward pass.

Forward mode would have to be repeated once for each variable. The price of reverse mode is memory: the intermediate values of the forward pass have to be stored.

Leads to

Sources

How to cite

“Reverse-mode automatic differentiation.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/linnainmaa-reverse-mode-differentiation

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline