The History of AI

The neural network winter

How neural networks survived twenty years in the background, from Hopfield's physics to GPUs, and why they came back with AlexNet.

  1. The Hopfield network (April 1982)

    Hopfield, a physicist, presents a network that stores patterns as energy minima and recovers them from a fragment. It draws many physicists to neural networks.

  2. The Boltzmann machine (January 1985)

    Hinton and colleagues take the Hopfield network as their starting point and add hidden units and random decisions. The Boltzmann machine learns internal representations.

  3. Backpropagation by Rumelhart, Hinton, and Williams (October 1986)

    Meanwhile, Rumelhart, Hinton, and Williams popularize backpropagation, the method that trains networks with hidden layers by sending the error backward.

  4. LeCun's convolutional network (1989)

    LeCun uses backpropagation to train a convolutional network that reads handwritten zip codes on real mail.

  5. LeNet-5 and MNIST (1998)

    Ten years later, the same group presents LeNet-5 and the MNIST database. Their system reads checks commercially.

  6. Deep belief nets (July 2006)

    With networks out of the spotlight, Hinton stacks restricted Boltzmann machines and trains them one layer at a time. For the first time, a deep network is trained convincingly.

  7. CUDA (February 15, 2007)

    Meanwhile, Nvidia releases CUDA, and graphics cards can be programmed for general-purpose computing.

  8. Deep networks trained on GPUs (June 2009)

    Raina, Madhavan, and Ng move deep belief networks onto the GPU with CUDA. Training them drops from weeks to a day.

  9. ImageNet (June 2009)

    Meanwhile, Fei-Fei Li and collaborators assemble ImageNet, millions of hand-labeled images.

  10. DanNet (2011)

    At IDSIA, a LeNet-style convolutional network, deeper and trained on GPUs, beats humans in a traffic sign competition.

  11. AlexNet (September 30, 2012)

    AlexNet is trained on two GPUs with CUDA on ImageNet and wins the 2012 competition by a wide margin. Neural networks are back at the center of the field.

Open in the interactive timeline