The History of AI

LSTM

Era 3 · From the expert boom to the statistical turn (1980–2011) › Neural networks

Lineage
Neurons and networks
Kind
Publication

Who

Technical University of Munich, IDSIA

What happened

Sepp Hochreiter and Jürgen Schmidhuber publish "Long Short-Term Memory" in Neural Computation. They had presented the idea in a 1995 technical report and at the 1996 NIPS conference.

Why it mattered

A recurrent network with a memory cell and gates that decide what goes in, what is kept, and what comes out. It overcomes the vanishing gradient problem and remembers information for more than a thousand steps.

The full story

Recurrent networks process sequences, such as sentences or audio signals, by keeping a state that passes from one step to the next. In theory they can remember what they saw long before, but in practice they did not learn to. Hochreiter, in a 1991 diploma thesis supervised by Schmidhuber, analyzed why: when the error is backpropagated through many steps, the gradient is multiplied over and over and either vanishes or explodes.

The LSTM solves the problem with a memory cell through which the error can flow without fading, what the authors called the constant error carousel. Gates, small networks that learn when to open, control what information enters the cell and when it leaves. In 1999 Felix Gers, Schmidhuber, and Fred Cummins added the forget gate, which lets the cell empty itself, and that is the version that caught on.

For years the LSTM was a niche technique. From 2015 on it became the basis of Google's speech recognition, of its 2016 neural translation system, which according to the company cut translation errors by 60 percent, and of Apple's voice assistants.

It was the dominant architecture for sequences until the Transformer, from 2017 on, replaced it in most language tasks.

A rectangular cell crossed at the top by a horizontal line, the memory, which enters on the left and leaves on the right. The input arrives from below at three gates: the forget gate multiplies the memory, the input gate adds new information to it, and the output gate takes the memory and produces the cell's output.

The cell's memory runs from one step to the next almost unchanged; three learned gates decide what is forgotten, what is added, and what comes out. The forget gate was added in 1999.

Under the hood

In the version that caught on, with the 1999 forget gate, each step computes three gates with values between 0 and 1: forget f, input i, and output o, all from the current input xₜ and the previous output hₜ₋₁. The memory is updated with cₜ = f ⊙ cₜ₋₁ + i ⊙ g, where g is the proposed new content and ⊙ the elementwise product, and the output is hₜ = o ⊙ tanh(cₜ).

The memory passes from one step to the next through a sum, not a multiplication by a weight matrix. In a simple recurrent network, when the error is backpropagated, the gradient is multiplied at each step by the same matrix and tends to zero or blows up. In the LSTM, as long as the forget gate stays close to 1, the gradient passes through the cell almost intact: that is the constant error carousel. The 1997 paper had no forget gate, and the memory was always kept.

Sources

Original source

How to cite

“LSTM.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/lstm

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline