Backpropagation by Rumelhart, Hinton, and Williams
Era 3 · From the expert boom to the statistical turn (1980–2011) › Neural networks
- Lineage
- Neurons and networks
- Kind
- Popularization
Who
University of California, San Diego, Carnegie Mellon
What happened
David Rumelhart, Geoffrey Hinton, and Ronald Williams publish "Learning representations by back-propagating errors" in Nature. The same year they develop the method in a chapter of Parallel Distributed Processing.
Why it mattered
They popularize backpropagation and show that a network with hidden layers learns useful internal representations. It has been the method used to train nearly every neural network since.
The full story
The method for computing gradients backward had existed since Linnainmaa in 1970, Amari had trained a multilayer network by gradient descent in 1967, and Werbos had applied it to neural networks in 1982. David Parker and LeCun also arrived at it independently in 1985. What was missing was a convincing demonstration, to a wide audience, that with it a multilayer network learned something it had not been explicitly taught.
Rumelhart, a psychologist in San Diego, Hinton, and Williams did that with carefully chosen examples. In one, the network learned family relationships in two family trees, one English and one Italian, and its hidden units discovered on their own features such as each person's nationality or generation, without anyone pointing them out. The hidden layers, which the perceptron could not train, built their own representations.
The paper appeared in Nature, and the corresponding chapter was part of Parallel Distributed Processing, the two volumes by the group of Rumelhart and James McClelland that became the manifesto of connectionism. Together they gave neural networks a legitimacy they had not had since 1969.
Backpropagation is today the core of neural network training, from convolutional networks to the Transformer. The ACM cited this work in awarding Hinton the 2018 Turing Award, announced in 2019, and the debate over who deserves priority for the method remains open.
A network with two inputs, three hidden neurons, and one output. An upper arrow shows the signal moving forward to the output, where the error is computed as the difference between the output and the target. A lower arrow, in another color, shows the error returning layer by layer and correcting each weight, such as the highlighted weight w between a hidden neuron and the output.
Under the hood
A network computes its output layer by layer. In each neuron, z = Σ wᵢ·aᵢ + b and a = f(z), where f is a smooth function such as the sigmoid. The error measures the distance between the output and the target, for example E = ½·(y − t)².
To correct a weight, you need to know how much the error changes when it moves: the derivative ∂E/∂w. Backpropagation gets it with the chain rule. At the output, δ = (y − t)·f′(z). In a hidden neuron j, δⱼ = f′(zⱼ)·Σₖ wⱼₖ·δₖ: its share of the blame is the sum of the blame of the neurons it feeds, weighted by their connections. With those values, each weight is corrected by Δwᵢⱼ = −η·δⱼ·aᵢ, where η is the learning rate.
The practical key is cost: computing all the derivatives costs about as much as one forward pass, however many weights the network has. That is why the method scales to networks with billions of parameters.
Common misconceptions
What people often say: Rumelhart, Hinton, and Williams invented backpropagation in 1986.
In fact: Computing gradients backward is Linnainmaa's (1970), and Werbos applied it to neural networks in 1982. The 1986 paper showed that with it hidden layers learn useful representations, and it was the paper that spread the method.
What people often say: Backpropagation imitates the way the brain learns.
In fact: It is a mathematical method for computing derivatives. Biological neurons do not send an exact error back along the same connections, and how the brain learns is still an open research question.
Videos
- Geoffrey Hinton and Yann LeCun, 2018 ACM A.M. Turing Award Lecture "The Deep Learning Revolution" (opens in a new tab) (Association for Computing Machinery (ACM))
Builds on
Forerunners of backpropagation (October 1960), Reverse-mode automatic differentiation (1970), Werbos's dissertation (1974)
Independent convergence
Rumelhart arrived at the method independently around 1982, and the 1986 paper does not cite the earlier work
Leads to
LeCun's convolutional network (1989)
Documented direct influence
The 1989 network is trained with backpropagation, as the paper's title says
Threads that pass through here
- The neural network winter (step 3 of 11)
Sources
Original source
- Learning representations by back-propagating errors. Nature, 323(6088), pp. 533-536 (opens in a new tab). David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, 1986
Scholarship and context
Further reading
How to cite
“Backpropagation by Rumelhart, Hinton, and Williams.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/backpropagation-1986
This entry was last reviewed on November 13, 2026.