Vision Transformer
Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models
- Lineage
- Neurons and networks
- Kind
- Publication
Who
- Alexey Dosovitskiy
- Lucas Beyer
- Alexander Kolesnikov
- Dirk Weissenborn
- Xiaohua Zhai
- Thomas Unterthiner
- Mostafa Dehghani
- Matthias Minderer
- Georg Heigold
- Sylvain Gelly
- Jakob Uszkoreit
- Neil Houlsby
What happened
Alexey Dosovitskiy and eleven coauthors post "An Image is Worth 16x16 Words" to arXiv on October 22, 2020. They apply a standard Transformer, with no convolutions, to image classification. The final version is presented at ICLR 2021.
Why it mattered
The architecture built for language works for vision too: pretrained on enough data, the Vision Transformer matches or beats the best convolutional networks while using substantially less training compute.
The full story
Since AlexNet in 2012, computer vision had run on convolutional networks, designed specifically for images. The Transformer had pushed recurrent networks out of language, but in vision, attention was used alongside convolutions or in place of some of their parts, never without them.
The Vision Transformer tries the simplest thing possible. It cuts the image into fixed-size patches (16 by 16 pixels, hence the title), turns each patch into a vector, and hands the sequence to a standard Transformer as if every patch were a word. There is not a single convolution.
Trained only on mid-sized datasets like ImageNet, the model came in a few points below a ResNet convolutional network of similar size. The authors put this down to the Transformer lacking the built-in assumptions about images that convolutions carry. The picture changes with more data. Pretrained on datasets of 14 million to 300 million images (ImageNet-21k and JFT-300M, an internal Google dataset) and then fine-tuned on mid-sized and small benchmarks, the Vision Transformer matched or beat the best convolutional networks and needed substantially less compute to train.
The final version of the paper is the one presented at ICLR 2021, and the code and pretrained models were released on GitHub. With this work, the Transformer became a general-purpose architecture, no longer just a text model.
Builds on
The Transformer (June 12, 2017)
Documented direct influence
The paper applies "a pure transformer" directly "to sequences of image patches"
Leads to
Sora (February 15, 2024)
Inherited, with no proven direct link
Sora is a diffusion model that works on a compressed representation of video, split into patches like the images in the Vision Transformer
Threads that pass through here
- From diffusion to world models (step 2 of 7)
Sources
Original source
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv:2010.11929) (opens in a new tab). Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, 2020
- p. 3 (opens in a new tab) Figure 1: the image is cut into fixed-size patches that the Transformer reads as if they were words.
Further reading
How to cite
“Vision Transformer.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/vision-transformer
This entry was last reviewed on November 13, 2026.