The History of AI

A multilayer network trained with stochastic gradient descent

Era 2 · AI is born (1943–1979) › Neural networks and learning

Lineage
Neurons and networks
Kind
Implementation

Who

What happened

Shun'ichi Amari publishes "A Theory of Adaptive Pattern Classifiers" in the IEEE Transactions on Electronic Computers, proposing to train every layer of a classifier at once by stochastic gradient descent.

Why it mattered

According to Schmidhuber's history of deep learning, it is the first multilayer network trained end to end with stochastic gradient descent. With a student, Saito, Amari trains a five-layer network that learns internal representations.

The full story

Rosenblatt's and Widrow's methods adjusted a single layer. Amari, a Japanese mathematician, posed the problem in general terms: a classifier with several layers of units and a loss function that measures its errors. If the units have a smooth output instead of an abrupt threshold, the loss is differentiable with respect to every weight, and it can be reduced by moving each weight in the direction of its gradient.

Instead of computing the gradient over all the examples at once, Amari updated the weights after each example, by a small step. This is stochastic gradient descent, which Robbins and Monro had proposed in 1951 as an approximation method for noisy observations and which Amari carried to every layer of a network. It is the procedure used to train almost every neural network today, and the paper analyzes when it converges.

With Saito, Amari put the idea into practice: a five-layer network, two of whose layers were adjustable, learned to classify patterns that cannot be separated by a straight line, which forces the inner layers to build their own representations. That was exactly what the single-layer perceptron could not do.

The work appeared in an engineering journal at an unfavorable moment, shortly before Minsky and Papert's book. Amari moved on to other fields, such as information geometry, and the contribution to deep learning was recognized much later.

Videos

Builds on

  • Stochastic approximation (September 1951)

    Inherited, with no proven direct link

    Stochastic gradient descent is the Robbins-Monro method applied to the gradient of an error function; Amari uses it to train every layer

Sources

How to cite

“A multilayer network trained with stochastic gradient descent.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/amari-stochastic-gradient-descent

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline