The History of AI

Deep networks trained on GPUs

Era 3 · From the expert boom to the statistical turn (1980–2011) › Hardware and data

Lineage
Machines and computation
Kind
Implementation

Who

Stanford University

What happened

Rajat Raina, Anand Madhavan, and Andrew Ng of Stanford show at ICML that GPUs train deep networks up to 70 times faster than a dual-core processor: a four-layer network with 100 million parameters trains in a day instead of weeks.

Why it mattered

It shows that cheap graphics hardware makes it feasible to train large deep networks on lots of data, one of the conditions for the takeoff of deep learning.

The full story

In 2006 Hinton had shown that deep networks could be trained layer by layer without labels. The problem was time: on the processors of the day, the interesting models, with many parameters and lots of data, took weeks to train, so researchers settled for small ones.

Raina, Madhavan, and Ng asked whether GPUs, which could now be programmed with CUDA, would change that. They adapted two unsupervised learning methods of the time, deep belief nets and sparse coding, to exploit the graphics card's parallelism, and proposed general principles for spreading this kind of learning across a GPU's thousands of threads.

The results were clear. Deep belief nets trained up to 70 times faster than on a dual-core processor, and a four-layer network with 100 million parameters went from weeks to about a day. Sparse coding ran 5 to 15 times faster.

The paper's message was that model size was no longer limited by the researcher's patience. Within a few years, training on GPUs became the norm in deep learning.

Builds on

  • Deep belief nets (July 2006)

    Documented direct influence

    Raina, Madhavan, and Ng speed up precisely deep belief networks on GPUs and cite Hinton's work

  • CUDA (February 15, 2007)

    Documented direct influence

    The paper programs Nvidia GPUs with CUDA to spread learning across thousands of threads

Threads that pass through here

Sources

How to cite

“Deep networks trained on GPUs.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/deep-networks-trained-on-gpus

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline