The History of AI

The Transformer

Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models

Lineage
Neurons and networks
Kind
Publication

Who

Google Brain, Google Research

What happened

Eight researchers at Google Brain and Google Research post "Attention Is All You Need" to arXiv on June 12, 2017; the paper later appears in the NIPS 2017 proceedings. It introduces the Transformer, a translation architecture built on attention alone.

Why it mattered

It does away with recurrence. The model reads every word in a sentence at once, so it trains far faster, and it became the architecture underneath BERT, the GPT series, and most large language models since.

The full story

In 2017, neural machine translation ran on sequence-to-sequence models. One recurrent network read the source sentence a word at a time, a second one wrote out the translation, and an attention mechanism let the second network look back at whichever part of the source it needed at each step. The bottleneck was the recurrence itself. Every step waits on the one before it, and as the authors point out, that sequential nature makes it impossible to split the computation for a single example across many processors.

The Transformer keeps the attention and drops the recurrence. Every word is compared with every other word in the sentence simultaneously, through several attention "heads" running in parallel (eight in the base model), each free to pick up on a different kind of relationship. The encoder and the decoder are each a stack of six identical layers. Because the model no longer walks through the sentence in order, it is told each word's position as one more input.

It was better and cheaper. On the WMT 2014 English-to-German translation task it scored 28.4 BLEU, and on English-to-French it reached 41.8, a record for a single model. The large version trained in three and a half days on eight GPUs.

Within a year, GPT-1 and BERT were both built on the Transformer, and in 2020 the Vision Transformer carried it over to images. As of 2026 it has more than 250,000 citations and ranks among the ten most cited papers of the twenty-first century. All eight authors left Google in the years that followed, to found companies or join others; Noam Shazeer came back in 2024 to co-lead Gemini.

Two columns of stacked boxes. The left one, the encoder, holds multi-head attention, add and normalize, a feed-forward network, and another add and normalize. The right one, the decoder, adds masked attention and an attention layer over the encoder, which an arrow from the left column feeds into. On top sit a linear layer and a softmax.

On the left, the encoder reads the sentence; on the right, the decoder writes the output one word at a time, consulting the encoder as it goes. Each column repeats its block N times, six in the paper.

Under the hood

The core operation is scaled dot-product attention: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. Every word produces a query (Q), a key (K), and a value (V), and how closely a query matches each key decides how much the corresponding value contributes. Dividing by √d_k keeps the dot products from growing with the dimension and saturating the softmax.

Multi-head attention runs eight of these in parallel, each with its own projections, and concatenates the results. With no recurrence, word order comes in through a positional encoding built from sines and cosines and added to the input vectors.

The base model has six layers in the encoder and six in the decoder, working with 512-dimensional vectors.

Common misconceptions

  • What people often say: The Transformer invented attention.

    In fact: Attention in machine translation dates to 2014, from Bahdanau, Cho, and Bengio. What was new in the Transformer was dropping recurrent networks entirely and building everything out of attention.

Videos

Builds on

  • Sequence to sequence (June 3, 2014), The attention mechanism (September 1, 2014)

    Documented direct influence

    The Transformer drops recurrence and keeps attention

  • ResNet (December 10, 2015)

    Documented direct influence

    The Transformer employs "a residual connection" around each sub-layer of the encoder and the decoder, citing He et al. (section 3.1)

Leads to

  • ELMo, GPT-1, and BERT (February 15, 2018)

    Documented direct influence

    GPT-1 combines Transformers and unsupervised pretraining, and BERT's architecture is "a multi-layer bidirectional Transformer encoder" based on the original implementation by Vaswani et al.

  • Vision Transformer (October 22, 2020)

    Documented direct influence

    The paper applies "a pure transformer" directly "to sequences of image patches"

  • AlphaFold 2 (November 2020)

    Inherited, with no proven direct link

    AlphaFold 2's Evoformer blocks are built on attention and its structure module uses an "equivariant transformer"; the paper does not present its design as a direct adaptation of the Transformer

Threads that pass through here

Sources

Original source

How to cite

“The Transformer.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/transformer

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline