The History of AI

The attention mechanism

Era 4 · Deep learning and foundation models (2012–2021) › Deep learning

Lineage
Neurons and networks
Kind
Publication

Who

Université de Montréal

What happened

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio post on arXiv a neural translator that, as it writes each word, learns to search for the parts of the source sentence that help. It is presented as an oral paper at ICLR 2015.

Why it mattered

The attention mechanism: at each step the model decides which part of the input to look at. It is the idea the Transformer will make its central component in 2017.

The full story

Sequence-to-sequence models summed up the whole source sentence in a single fixed-size vector. Bahdanau, Cho, and Bengio argued that this vector was a bottleneck: a long sentence held more than would fit in it to translate the sentence well.

Their proposal was to stop squeezing everything into one vector. The encoder produces a representation for each word of the source sentence, and the decoder, before writing each word of the translation, performs a soft search: it computes how relevant each of those representations is and combines them with those weights. Nobody tells it which word corresponds to which; the alignment is learned along with the translation.

On English-to-French translation the model reached results comparable to those of the best phrase-based statistical system. What is more, the attention weights could be plotted, and the alignments they showed matched a human translator's intuition.

The idea that a network can decide for itself which part of the input to look at turned out to be more general than translation. Three years later the Transformer did away with recurrent networks and kept only attention.

Two rows of four boxes, h1 to h4, one per word of "the cat sat down." In the top row, the last box leads to a single vector c and from there to the translator. In the bottom row, the four boxes connect to a "next word" box by lines of different thickness, with weights of 0.1, 0.7, 0.15, and 0.05; the thickest comes from "cat."

Above, the encoder compresses the whole sentence into a single vector c. Below, with attention, each word of the translation weighs all the encoder states. The α weights are an example.

Under the hood

To write word i, the decoder computes a context vector of its own: cᵢ = Σⱼ αᵢⱼ hⱼ, the average of the encoder states hⱼ weighted by the weights αᵢⱼ.

The weights come from a small network that scores how well each state hⱼ fits the decoder's previous state; a softmax turns those scores into weights that add up to 1. The whole network, scoring included, is trained at once. The encoder is a bidirectional recurrent network, so each hⱼ sums up word j together with what lies on either side of it.

Builds on

  • Sequence to sequence (June 3, 2014)

    Documented direct influence

    The paper argues that the fixed-length vector is a bottleneck in the encoder-decoder architecture and proposes extending it so the model can search on its own for the parts of the source sentence that matter

Leads to

  • The Transformer (June 12, 2017)

    Documented direct influence

    The Transformer drops recurrence and keeps attention

Threads that pass through here

Sources

How to cite

“The attention mechanism.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/bahdanau-attention

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline