Deep belief nets
Era 3 · From the expert boom to the statistical turn (1980–2011) › Neural networks
- Lineage
- Neurons and networks
- Kind
- Publication
Who
University of Toronto
What happened
Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh publish "A Fast Learning Algorithm for Deep Belief Nets" in Neural Computation, a method for training many-layered networks one layer at a time.
Why it mattered
Layer-by-layer pretraining, with no labels, makes it possible to train deep networks that previously failed to learn. It revives the field under the name deep learning.
The full story
In the early 2000s neural networks had been pushed into the background. Networks with many layers were hard to train with backpropagation from random weights, and kernel methods gave results just as good with stronger guarantees. Hinton, in Toronto, kept looking for a way to make deep networks learn.
The answer was not to train them all at once. Each layer is a restricted Boltzmann machine, with no connections between units in the same layer, and it is trained separately to model the activations of the layer below, with no need for labels. Once a layer has learned, its outputs serve as data for the next one. At the end, the whole network, which by now has reasonable weights, is fine-tuned with a slower method. This produced a network with three hidden layers that classified MNIST digits better than the best discriminative methods of the time, when none of them was given any information about the shape of the image.
That same year, Hinton and Ruslan Salakhutdinov showed in Science that the method reduced the dimensionality of data better than principal component analysis. Together, these papers convinced the community that deep networks could be trained, and the label deep learning, which had been used before, caught on.
Layer-by-layer pretraining was abandoned a few years later, once it became clear that with more data, GPUs, and better activation functions, networks could be trained directly. But it was the key that opened the door again.
Builds on
The Boltzmann machine (January 1985)
Documented direct influence
Each layer of the deep belief network is a restricted Boltzmann machine trained on its own
Leads to
Deep networks trained on GPUs (June 2009)
Documented direct influence
Raina, Madhavan, and Ng speed up precisely deep belief networks on GPUs and cite Hinton's work
Deep networks in speech recognition (August 2011)
Documented direct influence
The recognizer's networks are pretrained layer by layer as deep belief networks
Threads that pass through here
- The neural network winter (step 6 of 11)
Sources
Original source
- A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18(7), pp. 1527-1554 (opens in a new tab). Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh, 2006
Scholarship and context
Further reading
How to cite
“Deep belief nets.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/deep-belief-nets
This entry was last reviewed on November 13, 2026.