The History of AI

Temporal differences

Era 3 · From the expert boom to the statistical turn (1980–2011) › Reinforcement learning

Lineage
Data and learning
Kind
Publication

Who

GTE Laboratories

What happened

Richard Sutton publishes "Learning to Predict by the Methods of Temporal Differences" in the journal Machine Learning, formalizing and analyzing temporal-difference learning.

Why it mattered

A system learns to predict by correcting each estimate with the next one, without waiting for the final outcome. It is one of the pillars of reinforcement learning.

The full story

To learn to predict the outcome of a game, or Saturday's weather, the classic method waits until the outcome is known and then corrects the prediction. Sutton proposed something else: compare each prediction with the next. If on Monday the forecast for Saturday is a 50 percent chance of rain and on Tuesday it is 70 percent, the difference is already information, and Monday's prediction can be corrected without waiting for the weekend.

The idea had precedents, from Samuel's checkers program, which adjusted its evaluation of a position using that of later positions, to the models of animal conditioning that Sutton and Andrew Barto had been working on since the late 1970s at the University of Massachusetts. The 1988 paper turned it into a family of algorithms, TD(λ), with a parameter that sets how much weight distant predictions carry, and proved that in certain cases it converges and learns from less data than methods that wait for the end.

Temporal differences also turned out to be a good model of the brain: years later it was found that dopamine neurons respond much like the prediction error the algorithm uses.

The method lies at the heart of Q-learning, of TD-Gammon, and, combined with deep networks, of the system that learned to play Atari games from pixels. Sutton and Barto received the 2024 Turing Award, announced in 2025, for founding reinforcement learning.

Bar chart of the probability of rain on Saturday as estimated each day: 50 percent on Monday and 70 percent on Tuesday; on Saturday it rains. A short arrow leads from Tuesday back to Monday and represents the temporal-difference correction. A long dashed arrow runs from Saturday back to Monday and represents the classic method, which waits for the outcome.

On Monday the forecast for Saturday is a 50 percent chance of rain, and on Tuesday 70 percent: the difference is enough to correct Monday's prediction.

Under the hood

In the form used today, let V(s) be the prediction in state s. After moving to state s′ and receiving reward r, the temporal-difference error is δ = r + γ·V(s′) − V(s), and the prediction is corrected with V(s) ← V(s) + α·δ. α is the learning rate, and γ, between 0 and 1, is a discount that gives less weight to what lies further ahead.

That is what the rain example does with Monday's prediction: its target is not the final outcome but the next estimate. The method learns from one guess to the next.

TD(λ) moves between two extremes. With λ = 0 it looks only one step ahead; with λ = 1 it is equivalent to waiting for the final outcome. Intermediate values spread each error across recent states through a trace that fades over time.

Builds on

  • The law of effect (1898)

    Inherited, with no proven direct link

    The law of effect is the conceptual forerunner, with no direct line between the authors

  • Samuel's checkers program (1952)

    Documented direct influence

    The temporal-difference method grows out of Samuel's work. Sutton and Barto write that Samuel was the first to propose and implement a learning method built on temporal-difference ideas, in Samuel's checkers program

  • Dynamic Programming (1957)

    Documented direct influence

    Sutton and Barto trace one of the roots of reinforcement learning to dynamic programming and the Bellman equation; temporal-difference methods approximate that equation

  • The conditioned reflex (April 23, 1903)

    Inherited, with no proven direct link

    Sutton and Barto also develop temporal-difference learning as a model of classical conditioning, the phenomenon Pavlov described

Leads to

  • TD-Gammon (1992)

    Documented direct influence

    Tesauro applies the algorithm directly

  • Q-learning (1989)

    Documented direct influence

    According to Sutton and Barto, Q-learning brings together the temporal-difference thread and the optimal-control thread in 1989

  • Turing Award for Barto and Sutton (March 5, 2025)

    Documented direct influence

    The ACM honors the conceptual and algorithmic foundations of reinforcement learning, temporal-difference learning among them

Threads that pass through here

Sources

How to cite

“Temporal differences.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/sutton-temporal-differences

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline