Reverse-mode automatic differentiation
Era 2 · AI is born (1943–1979) › Neural networks and learning
- Lineage
- Neurons and networks
- Kind
- Theory
Who
University of Helsinki
What happened
In a master's thesis at the University of Helsinki, Seppo Linnainmaa presents a method for computing the accumulated effect of rounding errors that is the reverse mode of automatic differentiation. Part of the results appear in English in 1976, in the journal BIT.
Why it mattered
It is the modern, general form of backpropagation: it computes all the derivatives of a composite function in a single backward pass, at a cost of the same order as computing the function itself.
The full story
Linnainmaa was not thinking about neural networks but about a problem in numerical analysis. A program performs thousands of operations on rounded numbers, and each rounding introduces a small error. To know how much each one affects the final result, you need the derivative of the result with respect to every intermediate operation.
The solution was to run the calculation in reverse. The program runs forward once, storing its intermediate values, and then works back from the result to the beginning, applying the chain rule at each operation. That yields the derivatives with respect to every intermediate variable in a single pass, at a cost comparable to the original calculation, instead of repeating the calculation once per variable.
As Linnainmaa told Andreas Griewank, the idea came on a sunny afternoon in 1970 in a park in Copenhagen. Linnainmaa presented the method as a way of estimating rounding errors, not as a cheap way to compute gradients, and the work went almost unnoticed until Griewank got in touch in the late 1980s.
A neural network is a composite function like any other, and training it with backpropagation means applying this reverse mode. That is why Schmidhuber considers Linnainmaa the first to publish what is now called backpropagation.
A chain of computation running left to right, from x to a multiplication by w, a function f, and the result E. Below, dashed arrows run back from E to the left with the derivatives of E with respect to y, a, and w, each obtained from the next.
Under the hood
If the calculation is the chain a = w·x, y = f(a), E = g(y), the chain rule gives ∂E/∂w = g′(y) · f′(a) · x.
Reverse mode walks that chain from the end: it computes ∂E/∂y, uses it to get ∂E/∂a, and uses that to get ∂E/∂w. Each intermediate derivative is computed only once and serves every earlier variable. With many inputs and a single output, like the error of a network with millions of weights, the whole gradient comes at a cost of a few times that of the forward pass.
Forward mode would have to be repeated once for each variable. The price of reverse mode is memory: the intermediate values of the forward pass have to be stored.
Leads to
Backpropagation by Rumelhart, Hinton, and Williams (October 1986)
Independent convergence
Rumelhart arrived at the method independently around 1982, and the 1986 paper does not cite the earlier work
Sources
Original source
- Taylor expansion of the accumulated rounding error. BIT, 16(2), pp. 146-160 (opens in a new tab). Seppo Linnainmaa, 1976
Scholarship and context
How to cite
“Reverse-mode automatic differentiation.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/linnainmaa-reverse-mode-differentiation
This entry was last reviewed on November 13, 2026.