The History of AI

ResNet

Era 4 · Deep learning and foundation models (2012–2021) › Deep learning

Lineage
Neurons and networks
Kind
Publication

Who

Microsoft Research Asia

What happened

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun of Microsoft Research Asia publish residual networks: networks up to 152 layers deep, eight times deeper than earlier ones, which win the 2015 ImageNet classification competition with an error of 3.57 percent.

Why it mattered

Residual connections make it possible to train networks hundreds of layers deep. They become a standard part of almost every later deep network, the Transformer included.

The full story

After AlexNet, vision networks grew ever deeper, and those of 2014 reached about twenty layers. But adding layers stopped helping: past a certain point, a deeper network trained worse than a shallower one, even though in theory it could do the same job.

In May 2015 the highway networks of Srivastava, Greff, and Schmidhuber, at the Swiss lab IDSIA, had already trained networks hundreds of layers deep with connections regulated by gates. Kaiming He and colleagues, at Microsoft's Beijing lab, simplified the idea and changed what each block of layers learns. Instead of learning a complete transformation, a block learns only the difference, the residual, with respect to its input, which is added back through a shortcut. If a block has nothing to contribute, it only needs to learn a residual close to zero. That way the signal and the gradients pass through very deep networks without degrading.

On ImageNet they evaluated networks up to 152 layers deep, eight times deeper than the VGG networks and yet of lower complexity, and an ensemble of them achieved an error of 3.57 percent, first place in ILSVRC 2015 classification. With the same networks they also won ImageNet detection and localization and COCO detection and segmentation. On CIFAR-10 they tested networks with more than a thousand layers.

The paper was published at CVPR 2016. Residual connections are in almost every deep architecture that came afterward.

An input x passes down through two "weight layer" boxes, with a ReLU between them, to a circle with a plus sign. A colored line also leaves x, goes around the layers, and reaches the same circle. Below it is the output, F(x) + x.

A residual block: two layers compute a correction F(x), and a shortcut adds the input x to it unchanged.

Under the hood

A residual block computes y = F(x) + x: the layers only have to learn the difference F(x) between the desired output and the input. If the best thing is to change nothing, it is enough to drive F to zero, and that is easier than getting several nonlinear layers to reproduce their input.

The shortcut adds no parameters. Without shortcuts, the authors found that a 34-layer network had higher training error than an 18-layer one; with them, the 34-layer network beat the 18-layer one, and they were able to go to 152 layers on ImageNet.

Leads to

  • The Transformer (June 12, 2017)

    Documented direct influence

    The Transformer employs "a residual connection" around each sub-layer of the encoder and the decoder, citing He et al. (section 3.1)

Threads that pass through here

Sources

Original source

Scholarship and context

How to cite

“ResNet.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/resnet

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline