Stochastic approximation
Era 2 · AI is born (1943–1979) › Neural networks and learning
- Lineage
- Data and learning
- Kind
- Publication
Who
University of North Carolina
What happened
The statisticians Herbert Robbins and Sutton Monro publish "A Stochastic Approximation Method" in The Annals of Mathematical Statistics, a method for finding an unknown value from noisy measurements.
Why it mattered
It proves that you can reach the solution by correcting an estimate step by step with noisy observations, using smaller and smaller steps. Applied to the gradient of an error function, it is stochastic gradient descent.
The full story
Many problems in statistics and engineering call for finding the point at which a function reaches a certain level, when the function cannot be computed and can only be measured with error.
Robbins and Monro proposed a recursive procedure. Start from an estimate, take a measurement, correct the estimate in proportion to the difference between what was measured and what was sought, and repeat. The key lies in the step size: the steps have to shrink, but not so fast that the procedure stalls before it arrives. Under that condition, they proved, the estimate converges to the target value even though each measurement is noisy.
The following year, Jack Kiefer and Jacob Wolfowitz adapted the idea to search for the maximum of a function, estimating its slope from differences between measurements.
When what is being corrected is a model's parameters and the measurement is the gradient of its error on a single example, the Robbins-Monro method is stochastic gradient descent. Amari applied it to every layer of a network in 1967, and it is the procedure used to train almost every neural network today.
Leads to
A multilayer network trained with stochastic gradient descent (June 1967)
Inherited, with no proven direct link
Stochastic gradient descent is the Robbins-Monro method applied to the gradient of an error function; Amari uses it to train every layer
Sources
Original source
- A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), pp. 400-407 (opens in a new tab). Herbert Robbins and Sutton Monro, 1951
Further reading
How to cite
“Stochastic approximation.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/robbins-monro-stochastic-approximation
This entry was last reviewed on November 13, 2026.