A multilayer network trained with stochastic gradient descent
Era 2 · AI is born (1943–1979) › Neural networks and learning
- Lineage
- Neurons and networks
- Kind
- Implementation
Who
What happened
Shun'ichi Amari publishes "A Theory of Adaptive Pattern Classifiers" in the IEEE Transactions on Electronic Computers, proposing to train every layer of a classifier at once by stochastic gradient descent.
Why it mattered
According to Schmidhuber's history of deep learning, it is the first multilayer network trained end to end with stochastic gradient descent. With a student, Saito, Amari trains a five-layer network that learns internal representations.
The full story
Rosenblatt's and Widrow's methods adjusted a single layer. Amari, a Japanese mathematician, posed the problem in general terms: a classifier with several layers of units and a loss function that measures its errors. If the units have a smooth output instead of an abrupt threshold, the loss is differentiable with respect to every weight, and it can be reduced by moving each weight in the direction of its gradient.
Instead of computing the gradient over all the examples at once, Amari updated the weights after each example, by a small step. This is stochastic gradient descent, which Robbins and Monro had proposed in 1951 as an approximation method for noisy observations and which Amari carried to every layer of a network. It is the procedure used to train almost every neural network today, and the paper analyzes when it converges.
With Saito, Amari put the idea into practice: a five-layer network, two of whose layers were adjustable, learned to classify patterns that cannot be separated by a straight line, which forces the inner layers to build their own representations. That was exactly what the single-layer perceptron could not do.
The work appeared in an engineering journal at an unfavorable moment, shortly before Minsky and Papert's book. Amari moved on to other fields, such as information geometry, and the contribution to deep learning was recognized much later.
Videos
- Shun-ichi Amari (opens in a new tab) (International Neural Network Society (INNS))
Builds on
Stochastic approximation (September 1951)
Inherited, with no proven direct link
Stochastic gradient descent is the Robbins-Monro method applied to the gradient of an error function; Amari uses it to train every layer
Sources
Original source
- A Theory of Adaptive Pattern Classifiers. IEEE Transactions on Electronic Computers, EC-16(3), pp. 299-307 (opens in a new tab). Shun'ichi Amari, 1967
- A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), pp. 400-407 (opens in a new tab). Herbert Robbins and Sutton Monro, 1951
Scholarship and context
- Annotated History of Modern AI and Deep Learning (arXiv:2212.11279) (opens in a new tab). Jürgen Schmidhuber, 2022
How to cite
“A multilayer network trained with stochastic gradient descent.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/amari-stochastic-gradient-descent
This entry was last reviewed on November 13, 2026.