The Transformer
Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models
- Lineage
- Neurons and networks
- Kind
- Publication
Who
- Ashish Vaswani
- Noam Shazeer
- Niki Parmar
- Jakob Uszkoreit
- Llion Jones
- Aidan N. Gomez
- Łukasz Kaiser
- Illia Polosukhin
Google Brain, Google Research
What happened
Eight researchers at Google Brain and Google Research post "Attention Is All You Need" to arXiv on June 12, 2017; the paper later appears in the NIPS 2017 proceedings. It introduces the Transformer, a translation architecture built on attention alone.
Why it mattered
It does away with recurrence. The model reads every word in a sentence at once, so it trains far faster, and it became the architecture underneath BERT, the GPT series, and most large language models since.
The full story
In 2017, neural machine translation ran on sequence-to-sequence models. One recurrent network read the source sentence a word at a time, a second one wrote out the translation, and an attention mechanism let the second network look back at whichever part of the source it needed at each step. The bottleneck was the recurrence itself. Every step waits on the one before it, and as the authors point out, that sequential nature makes it impossible to split the computation for a single example across many processors.
The Transformer keeps the attention and drops the recurrence. Every word is compared with every other word in the sentence simultaneously, through several attention "heads" running in parallel (eight in the base model), each free to pick up on a different kind of relationship. The encoder and the decoder are each a stack of six identical layers. Because the model no longer walks through the sentence in order, it is told each word's position as one more input.
It was better and cheaper. On the WMT 2014 English-to-German translation task it scored 28.4 BLEU, and on English-to-French it reached 41.8, a record for a single model. The large version trained in three and a half days on eight GPUs.
Within a year, GPT-1 and BERT were both built on the Transformer, and in 2020 the Vision Transformer carried it over to images. As of 2026 it has more than 250,000 citations and ranks among the ten most cited papers of the twenty-first century. All eight authors left Google in the years that followed, to found companies or join others; Noam Shazeer came back in 2024 to co-lead Gemini.
Two columns of stacked boxes. The left one, the encoder, holds multi-head attention, add and normalize, a feed-forward network, and another add and normalize. The right one, the decoder, adds masked attention and an attention layer over the encoder, which an arrow from the left column feeds into. On top sit a linear layer and a softmax.
Under the hood
The core operation is scaled dot-product attention: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. Every word produces a query (Q), a key (K), and a value (V), and how closely a query matches each key decides how much the corresponding value contributes. Dividing by √d_k keeps the dot products from growing with the dimension and saturating the softmax.
Multi-head attention runs eight of these in parallel, each with its own projections, and concatenates the results. With no recurrence, word order comes in through a positional encoding built from sines and cosines and added to the input vectors.
The base model has six layers in the encoder and six in the decoder, working with 512-dimensional vectors.
Common misconceptions
What people often say: The Transformer invented attention.
In fact: Attention in machine translation dates to 2014, from Bahdanau, Cho, and Bengio. What was new in the Transformer was dropping recurrent networks entirely and building everything out of attention.
Videos
Builds on
Sequence to sequence (June 3, 2014), The attention mechanism (September 1, 2014)
Documented direct influence
The Transformer drops recurrence and keeps attention
ResNet (December 10, 2015)
Documented direct influence
The Transformer employs "a residual connection" around each sub-layer of the encoder and the decoder, citing He et al. (section 3.1)
Leads to
ELMo, GPT-1, and BERT (February 15, 2018)
Documented direct influence
GPT-1 combines Transformers and unsupervised pretraining, and BERT's architecture is "a multi-layer bidirectional Transformer encoder" based on the original implementation by Vaswani et al.
Vision Transformer (October 22, 2020)
Documented direct influence
The paper applies "a pure transformer" directly "to sequences of image patches"
AlphaFold 2 (November 2020)
Inherited, with no proven direct link
AlphaFold 2's Evoformer blocks are built on attention and its structure module uses an "equivariant transformer"; the paper does not present its design as a direct adaptation of the Transformer
Threads that pass through here
- From attention to the Transformer (step 4 of 4)
- From protein to Nobel (step 1 of 4)
Sources
Original source
- Attention Is All You Need (arXiv:1706.03762) (opens in a new tab). Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, 2017
- p. 3 (opens in a new tab) Figure 1: the full architecture, with the encoder on the left and the decoder on the right.
- p. 4 (opens in a new tab) Figure 2: scaled dot-product attention and multi-head attention, the piece that replaces recurrence.
Scholarship and context
- Attention is All you Need. Advances in Neural Information Processing Systems 30 (NIPS 2017) (opens in a new tab). Ashish Vaswani et al., 2017
How to cite
“The Transformer.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/transformer
This entry was last reviewed on November 13, 2026.