The History of AI

Backpropagation by Rumelhart, Hinton, and Williams

Era 3 · From the expert boom to the statistical turn (1980–2011) › Neural networks

Lineage
Neurons and networks
Kind
Popularization

Who

University of California, San Diego, Carnegie Mellon

What happened

David Rumelhart, Geoffrey Hinton, and Ronald Williams publish "Learning representations by back-propagating errors" in Nature. The same year they develop the method in a chapter of Parallel Distributed Processing.

Why it mattered

They popularize backpropagation and show that a network with hidden layers learns useful internal representations. It has been the method used to train nearly every neural network since.

The full story

The method for computing gradients backward had existed since Linnainmaa in 1970, Amari had trained a multilayer network by gradient descent in 1967, and Werbos had applied it to neural networks in 1982. David Parker and LeCun also arrived at it independently in 1985. What was missing was a convincing demonstration, to a wide audience, that with it a multilayer network learned something it had not been explicitly taught.

Rumelhart, a psychologist in San Diego, Hinton, and Williams did that with carefully chosen examples. In one, the network learned family relationships in two family trees, one English and one Italian, and its hidden units discovered on their own features such as each person's nationality or generation, without anyone pointing them out. The hidden layers, which the perceptron could not train, built their own representations.

The paper appeared in Nature, and the corresponding chapter was part of Parallel Distributed Processing, the two volumes by the group of Rumelhart and James McClelland that became the manifesto of connectionism. Together they gave neural networks a legitimacy they had not had since 1969.

Backpropagation is today the core of neural network training, from convolutional networks to the Transformer. The ACM cited this work in awarding Hinton the 2018 Turing Award, announced in 2019, and the debate over who deserves priority for the method remains open.

A network with two inputs, three hidden neurons, and one output. An upper arrow shows the signal moving forward to the output, where the error is computed as the difference between the output and the target. A lower arrow, in another color, shows the error returning layer by layer and correcting each weight, such as the highlighted weight w between a hidden neuron and the output.

The signal travels forward through the network; the error travels back, and each weight is corrected according to its share of the blame.

Under the hood

A network computes its output layer by layer. In each neuron, z = Σ wᵢ·aᵢ + b and a = f(z), where f is a smooth function such as the sigmoid. The error measures the distance between the output and the target, for example E = ½·(y − t)².

To correct a weight, you need to know how much the error changes when it moves: the derivative ∂E/∂w. Backpropagation gets it with the chain rule. At the output, δ = (y − t)·f′(z). In a hidden neuron j, δⱼ = f′(zⱼ)·Σₖ wⱼₖ·δₖ: its share of the blame is the sum of the blame of the neurons it feeds, weighted by their connections. With those values, each weight is corrected by Δwᵢⱼ = −η·δⱼ·aᵢ, where η is the learning rate.

The practical key is cost: computing all the derivatives costs about as much as one forward pass, however many weights the network has. That is why the method scales to networks with billions of parameters.

Common misconceptions

  • What people often say: Rumelhart, Hinton, and Williams invented backpropagation in 1986.

    In fact: Computing gradients backward is Linnainmaa's (1970), and Werbos applied it to neural networks in 1982. The 1986 paper showed that with it hidden layers learn useful representations, and it was the paper that spread the method.

  • What people often say: Backpropagation imitates the way the brain learns.

    In fact: It is a mathematical method for computing derivatives. Biological neurons do not send an exact error back along the same connections, and how the brain learns is still an open research question.

Videos

Builds on

Leads to

Threads that pass through here

Sources

Original source

How to cite

“Backpropagation by Rumelhart, Hinton, and Williams.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/backpropagation-1986

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline