The History of AI

Sequence to sequence

Era 4 · Deep learning and foundation models (2012–2021) › Deep learning

Lineage
Neurons and networks
Kind
Publication

Who

Université de Montréal, Google

What happened

Two 2014 papers propose translating with two recurrent networks, one that sums up the source sentence in a vector and another that writes the translation from it: one by Kyunghyun Cho and colleagues in Montreal (June) and one by Ilya Sutskever, Oriol Vinyals, and Quoc Le at Google (September).

Why it mattered

A single network, trained end to end, turns one sequence into another of a different length. It is the foundation of neural machine translation.

The full story

Machine translation in 2014 was statistical: tables of equivalent phrases and separate models for word order and fluency, tuned over years. Neural networks were used as auxiliary components inside that system.

In June, Cho, Bengio, and colleagues proposed a recurrent encoder-decoder: one network compresses the source sentence into a fixed-length vector and another generates the target sentence from it, and the two are trained together. They used it to score phrase pairs inside a statistical translator, and it improved the results.

In September, Sutskever, Vinyals, and Le took the idea all the way with multilayer LSTM networks that translated on their own. On the WMT 2014 English-to-French task they scored 34.8 BLEU, against 33.3 for a phrase-based statistical system. They also found a simple trick: reversing the order of the words in the source sentence improved the results considerably, because it created many short-range dependencies between the two sentences and made training easier.

The fixed-size vector was the weak point, because long sentences lost information. That limit, already visible in Cho's model, led in September to the attention mechanism, which appeared on arXiv nine days before the Sutskever paper. In 2016 Google rebuilt its translator on this line of work.

Builds on

  • IBM's statistical machine translation (1988)

    Inherited, with no proven direct link

    Sequence-to-sequence neural translation is measured against phrase-based statistical translation, the system it comes to replace

Leads to

  • The Transformer (June 12, 2017)

    Documented direct influence

    The Transformer drops recurrence and keeps attention

  • The attention mechanism (September 1, 2014)

    Documented direct influence

    The paper argues that the fixed-length vector is a bottleneck in the encoder-decoder architecture and proposes extending it so the model can search on its own for the parts of the source sentence that matter

Threads that pass through here

Sources

Original source

How to cite

“Sequence to sequence.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/sequence-to-sequence

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline