From attention to the Transformer
Attention first appears on top of a sequence-to-sequence model. The Transformer keeps the attention and drops the recurrence.
Sequence to sequence (June 3, 2014)
In June 2014, Cho and colleagues present an encoder-decoder that sums up the source sentence in a fixed-length vector. In September, Sutskever, Vinyals, and Le translate using networks of this kind alone.
The attention mechanism (September 1, 2014)
Bahdanau, Cho, and Bengio see that vector as a bottleneck. In September 2014, nine days before Sutskever's paper, they add an attention mechanism so the translator can look at whichever parts of the sentence it needs at each moment.
ResNet (December 10, 2015)
Residual networks add each block's own input back through a shortcut, which makes it possible to train networks hundreds of layers deep.
The Transformer (June 12, 2017)
The Transformer drops recurrence and keeps attention, with a residual connection around each sublayer.