The attention mechanism
Era 4 · Deep learning and foundation models (2012–2021) › Deep learning
- Lineage
- Neurons and networks
- Kind
- Publication
Who
Université de Montréal
What happened
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio post on arXiv a neural translator that, as it writes each word, learns to search for the parts of the source sentence that help. It is presented as an oral paper at ICLR 2015.
Why it mattered
The attention mechanism: at each step the model decides which part of the input to look at. It is the idea the Transformer will make its central component in 2017.
The full story
Sequence-to-sequence models summed up the whole source sentence in a single fixed-size vector. Bahdanau, Cho, and Bengio argued that this vector was a bottleneck: a long sentence held more than would fit in it to translate the sentence well.
Their proposal was to stop squeezing everything into one vector. The encoder produces a representation for each word of the source sentence, and the decoder, before writing each word of the translation, performs a soft search: it computes how relevant each of those representations is and combines them with those weights. Nobody tells it which word corresponds to which; the alignment is learned along with the translation.
On English-to-French translation the model reached results comparable to those of the best phrase-based statistical system. What is more, the attention weights could be plotted, and the alignments they showed matched a human translator's intuition.
The idea that a network can decide for itself which part of the input to look at turned out to be more general than translation. Three years later the Transformer did away with recurrent networks and kept only attention.
Two rows of four boxes, h1 to h4, one per word of "the cat sat down." In the top row, the last box leads to a single vector c and from there to the translator. In the bottom row, the four boxes connect to a "next word" box by lines of different thickness, with weights of 0.1, 0.7, 0.15, and 0.05; the thickest comes from "cat."
Under the hood
To write word i, the decoder computes a context vector of its own: cᵢ = Σⱼ αᵢⱼ hⱼ, the average of the encoder states hⱼ weighted by the weights αᵢⱼ.
The weights come from a small network that scores how well each state hⱼ fits the decoder's previous state; a softmax turns those scores into weights that add up to 1. The whole network, scoring included, is trained at once. The encoder is a bidirectional recurrent network, so each hⱼ sums up word j together with what lies on either side of it.
Builds on
Sequence to sequence (June 3, 2014)
Documented direct influence
The paper argues that the fixed-length vector is a bottleneck in the encoder-decoder architecture and proposes extending it so the model can search on its own for the parts of the source sentence that matter
Leads to
The Transformer (June 12, 2017)
Documented direct influence
The Transformer drops recurrence and keeps attention
Threads that pass through here
- From attention to the Transformer (step 2 of 4)
Sources
Original source
- Neural Machine Translation by Jointly Learning to Align and Translate (arXiv:1409.0473) (opens in a new tab). Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, 2014
Further reading
How to cite
“The attention mechanism.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/bahdanau-attention
This entry was last reviewed on November 13, 2026.