Deep networks trained on GPUs
Era 3 · From the expert boom to the statistical turn (1980–2011) › Hardware and data
- Lineage
- Machines and computation
- Kind
- Implementation
Who
Stanford University
What happened
Rajat Raina, Anand Madhavan, and Andrew Ng of Stanford show at ICML that GPUs train deep networks up to 70 times faster than a dual-core processor: a four-layer network with 100 million parameters trains in a day instead of weeks.
Why it mattered
It shows that cheap graphics hardware makes it feasible to train large deep networks on lots of data, one of the conditions for the takeoff of deep learning.
The full story
In 2006 Hinton had shown that deep networks could be trained layer by layer without labels. The problem was time: on the processors of the day, the interesting models, with many parameters and lots of data, took weeks to train, so researchers settled for small ones.
Raina, Madhavan, and Ng asked whether GPUs, which could now be programmed with CUDA, would change that. They adapted two unsupervised learning methods of the time, deep belief nets and sparse coding, to exploit the graphics card's parallelism, and proposed general principles for spreading this kind of learning across a GPU's thousands of threads.
The results were clear. Deep belief nets trained up to 70 times faster than on a dual-core processor, and a four-layer network with 100 million parameters went from weeks to about a day. Sparse coding ran 5 to 15 times faster.
The paper's message was that model size was no longer limited by the researcher's patience. Within a few years, training on GPUs became the norm in deep learning.
Builds on
Deep belief nets (July 2006)
Documented direct influence
Raina, Madhavan, and Ng speed up precisely deep belief networks on GPUs and cite Hinton's work
CUDA (February 15, 2007)
Documented direct influence
The paper programs Nvidia GPUs with CUDA to spread learning across thousands of threads
Threads that pass through here
- The neural network winter (step 8 of 11)
Sources
Original source
- Large-scale deep unsupervised learning using graphics processors. Proceedings of ICML 2009, pp. 873-880 (opens in a new tab). Rajat Raina, Anand Madhavan, and Andrew Y. Ng, 2009
Further reading
How to cite
“Deep networks trained on GPUs.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/deep-networks-trained-on-gpus
This entry was last reviewed on November 13, 2026.