ResNet
Era 4 · Deep learning and foundation models (2012–2021) › Deep learning
- Lineage
- Neurons and networks
- Kind
- Publication
Who
Microsoft Research Asia
What happened
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun of Microsoft Research Asia publish residual networks: networks up to 152 layers deep, eight times deeper than earlier ones, which win the 2015 ImageNet classification competition with an error of 3.57 percent.
Why it mattered
Residual connections make it possible to train networks hundreds of layers deep. They become a standard part of almost every later deep network, the Transformer included.
The full story
After AlexNet, vision networks grew ever deeper, and those of 2014 reached about twenty layers. But adding layers stopped helping: past a certain point, a deeper network trained worse than a shallower one, even though in theory it could do the same job.
In May 2015 the highway networks of Srivastava, Greff, and Schmidhuber, at the Swiss lab IDSIA, had already trained networks hundreds of layers deep with connections regulated by gates. Kaiming He and colleagues, at Microsoft's Beijing lab, simplified the idea and changed what each block of layers learns. Instead of learning a complete transformation, a block learns only the difference, the residual, with respect to its input, which is added back through a shortcut. If a block has nothing to contribute, it only needs to learn a residual close to zero. That way the signal and the gradients pass through very deep networks without degrading.
On ImageNet they evaluated networks up to 152 layers deep, eight times deeper than the VGG networks and yet of lower complexity, and an ensemble of them achieved an error of 3.57 percent, first place in ILSVRC 2015 classification. With the same networks they also won ImageNet detection and localization and COCO detection and segmentation. On CIFAR-10 they tested networks with more than a thousand layers.
The paper was published at CVPR 2016. Residual connections are in almost every deep architecture that came afterward.
An input x passes down through two "weight layer" boxes, with a ReLU between them, to a circle with a plus sign. A colored line also leaves x, goes around the layers, and reaches the same circle. Below it is the output, F(x) + x.
Under the hood
A residual block computes y = F(x) + x: the layers only have to learn the difference F(x) between the desired output and the input. If the best thing is to change nothing, it is enough to drive F to zero, and that is easier than getting several nonlinear layers to reproduce their input.
The shortcut adds no parameters. Without shortcuts, the authors found that a 34-layer network had higher training error than an 18-layer one; with them, the 34-layer network beat the 18-layer one, and they were able to go to 152 layers on ImageNet.
Leads to
The Transformer (June 12, 2017)
Documented direct influence
The Transformer employs "a residual connection" around each sub-layer of the encoder and the decoder, citing He et al. (section 3.1)
Threads that pass through here
- From attention to the Transformer (step 3 of 4)
Sources
Original source
- Deep Residual Learning for Image Recognition (arXiv:1512.03385) (opens in a new tab). Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, 2015
- Highway Networks (arXiv:1505.00387) (opens in a new tab). Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber, 2015
Scholarship and context
- Deep Residual Learning for Image Recognition. CVPR 2016, pp. 770-778 (opens in a new tab). Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, 2016
How to cite
“ResNet.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/resnet
This entry was last reviewed on November 13, 2026.