The History of AI

Deep belief nets

Era 3 · From the expert boom to the statistical turn (1980–2011) › Neural networks

Lineage
Neurons and networks
Kind
Publication

Who

University of Toronto

What happened

Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh publish "A Fast Learning Algorithm for Deep Belief Nets" in Neural Computation, a method for training many-layered networks one layer at a time.

Why it mattered

Layer-by-layer pretraining, with no labels, makes it possible to train deep networks that previously failed to learn. It revives the field under the name deep learning.

The full story

In the early 2000s neural networks had been pushed into the background. Networks with many layers were hard to train with backpropagation from random weights, and kernel methods gave results just as good with stronger guarantees. Hinton, in Toronto, kept looking for a way to make deep networks learn.

The answer was not to train them all at once. Each layer is a restricted Boltzmann machine, with no connections between units in the same layer, and it is trained separately to model the activations of the layer below, with no need for labels. Once a layer has learned, its outputs serve as data for the next one. At the end, the whole network, which by now has reasonable weights, is fine-tuned with a slower method. This produced a network with three hidden layers that classified MNIST digits better than the best discriminative methods of the time, when none of them was given any information about the shape of the image.

That same year, Hinton and Ruslan Salakhutdinov showed in Science that the method reduced the dimensionality of data better than principal component analysis. Together, these papers convinced the community that deep networks could be trained, and the label deep learning, which had been used before, caught on.

Layer-by-layer pretraining was abandoned a few years later, once it became clear that with more data, GPUs, and better activation functions, networks could be trained directly. But it was the key that opened the door again.

Builds on

  • The Boltzmann machine (January 1985)

    Documented direct influence

    Each layer of the deep belief network is a restricted Boltzmann machine trained on its own

Leads to

Threads that pass through here

Sources

Original source

How to cite

“Deep belief nets.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/deep-belief-nets

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline