Sequence to sequence
Era 4 · Deep learning and foundation models (2012–2021) › Deep learning
- Lineage
- Neurons and networks
- Kind
- Publication
Who
Université de Montréal, Google
What happened
Two 2014 papers propose translating with two recurrent networks, one that sums up the source sentence in a vector and another that writes the translation from it: one by Kyunghyun Cho and colleagues in Montreal (June) and one by Ilya Sutskever, Oriol Vinyals, and Quoc Le at Google (September).
Why it mattered
A single network, trained end to end, turns one sequence into another of a different length. It is the foundation of neural machine translation.
The full story
Machine translation in 2014 was statistical: tables of equivalent phrases and separate models for word order and fluency, tuned over years. Neural networks were used as auxiliary components inside that system.
In June, Cho, Bengio, and colleagues proposed a recurrent encoder-decoder: one network compresses the source sentence into a fixed-length vector and another generates the target sentence from it, and the two are trained together. They used it to score phrase pairs inside a statistical translator, and it improved the results.
In September, Sutskever, Vinyals, and Le took the idea all the way with multilayer LSTM networks that translated on their own. On the WMT 2014 English-to-French task they scored 34.8 BLEU, against 33.3 for a phrase-based statistical system. They also found a simple trick: reversing the order of the words in the source sentence improved the results considerably, because it created many short-range dependencies between the two sentences and made training easier.
The fixed-size vector was the weak point, because long sentences lost information. That limit, already visible in Cho's model, led in September to the attention mechanism, which appeared on arXiv nine days before the Sutskever paper. In 2016 Google rebuilt its translator on this line of work.
Builds on
IBM's statistical machine translation (1988)
Inherited, with no proven direct link
Sequence-to-sequence neural translation is measured against phrase-based statistical translation, the system it comes to replace
Leads to
The Transformer (June 12, 2017)
Documented direct influence
The Transformer drops recurrence and keeps attention
The attention mechanism (September 1, 2014)
Documented direct influence
The paper argues that the fixed-length vector is a bottleneck in the encoder-decoder architecture and proposes extending it so the model can search on its own for the parts of the source sentence that matter
Threads that pass through here
- From attention to the Transformer (step 1 of 4)
Sources
Original source
- Sequence to Sequence Learning with Neural Networks (arXiv:1409.3215) (opens in a new tab). Ilya Sutskever, Oriol Vinyals, and Quoc V. Le, 2014
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (arXiv:1406.1078) (opens in a new tab). Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, 2014
Further reading
How to cite
“Sequence to sequence.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/sequence-to-sequence
This entry was last reviewed on November 13, 2026.