CUDA
Era 3 · From the expert boom to the statistical turn (1980–2011) › Hardware and data
- Lineage
- Machines and computation
- Kind
- Implementation
Who
Nvidia
What happened
Nvidia releases the CUDA software development kit, a C programming interface that lets its graphics processing units (GPUs) be used for general computation and not just for drawing images.
Why it mattered
GPUs, with thousands of operations running in parallel, come within reach of any programmer. A few years later they become the engine of neural network training.
The full story
A graphics card computes the color of millions of pixels at once, and each pixel is worked out with the same operations on different data. In the early 2000s some researchers realized that this parallelism could serve other calculations, but to exploit it they had to disguise their problems as graphics operations.
At Stanford University, Ian Buck developed Brook, a language for programming GPUs as general-purpose processors. Nvidia hired Buck in 2004, and Buck worked with John Nickolls on what would become CUDA: an extension of the C language in which the programmer writes a function and the GPU runs it on thousands of threads at once.
Nvidia announced CUDA in November 2006, along with the GeForce 8800, and released the development kit in February 2007. The following year Nickolls, Buck, and others presented it as a parallel programming model that scales automatically to the number of cores on each card, which meant code could be written once and get faster with every new generation of hardware.
CUDA was not designed for artificial intelligence but for science and engineering. Yet training neural networks consists mostly of multiplying matrices, exactly the kind of computation a GPU does best, and over the following decade machine learning became the main reason to buy compute GPUs.
Leads to
AlexNet (September 30, 2012)
Documented direct influence
AlexNet is trained on GPUs with CUDA, on ImageNet
Deep networks trained on GPUs (June 2009)
Documented direct influence
The paper programs Nvidia GPUs with CUDA to spread learning across thousands of threads
Threads that pass through here
- Data, GPUs, and AlexNet (step 1 of 4)
- The neural network winter (step 7 of 11)
Sources
Original source
- Scalable Parallel Programming with CUDA. ACM Queue, 6(2), pp. 40-53 (opens in a new tab). John Nickolls, Ian Buck, Michael Garland, and Kevin Skadron, 2008
Further reading
How to cite
“CUDA.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/cuda
This entry was last reviewed on November 13, 2026.