Temporal differences
Era 3 · From the expert boom to the statistical turn (1980–2011) › Reinforcement learning
- Lineage
- Data and learning
- Kind
- Publication
Who
GTE Laboratories
What happened
Richard Sutton publishes "Learning to Predict by the Methods of Temporal Differences" in the journal Machine Learning, formalizing and analyzing temporal-difference learning.
Why it mattered
A system learns to predict by correcting each estimate with the next one, without waiting for the final outcome. It is one of the pillars of reinforcement learning.
The full story
To learn to predict the outcome of a game, or Saturday's weather, the classic method waits until the outcome is known and then corrects the prediction. Sutton proposed something else: compare each prediction with the next. If on Monday the forecast for Saturday is a 50 percent chance of rain and on Tuesday it is 70 percent, the difference is already information, and Monday's prediction can be corrected without waiting for the weekend.
The idea had precedents, from Samuel's checkers program, which adjusted its evaluation of a position using that of later positions, to the models of animal conditioning that Sutton and Andrew Barto had been working on since the late 1970s at the University of Massachusetts. The 1988 paper turned it into a family of algorithms, TD(λ), with a parameter that sets how much weight distant predictions carry, and proved that in certain cases it converges and learns from less data than methods that wait for the end.
Temporal differences also turned out to be a good model of the brain: years later it was found that dopamine neurons respond much like the prediction error the algorithm uses.
The method lies at the heart of Q-learning, of TD-Gammon, and, combined with deep networks, of the system that learned to play Atari games from pixels. Sutton and Barto received the 2024 Turing Award, announced in 2025, for founding reinforcement learning.
Bar chart of the probability of rain on Saturday as estimated each day: 50 percent on Monday and 70 percent on Tuesday; on Saturday it rains. A short arrow leads from Tuesday back to Monday and represents the temporal-difference correction. A long dashed arrow runs from Saturday back to Monday and represents the classic method, which waits for the outcome.
Under the hood
In the form used today, let V(s) be the prediction in state s. After moving to state s′ and receiving reward r, the temporal-difference error is δ = r + γ·V(s′) − V(s), and the prediction is corrected with V(s) ← V(s) + α·δ. α is the learning rate, and γ, between 0 and 1, is a discount that gives less weight to what lies further ahead.
That is what the rain example does with Monday's prediction: its target is not the final outcome but the next estimate. The method learns from one guess to the next.
TD(λ) moves between two extremes. With λ = 0 it looks only one step ahead; with λ = 1 it is equivalent to waiting for the final outcome. Intermediate values spread each error across recent states through a trace that fades over time.
Builds on
The law of effect (1898)
Inherited, with no proven direct link
The law of effect is the conceptual forerunner, with no direct line between the authors
Samuel's checkers program (1952)
Documented direct influence
The temporal-difference method grows out of Samuel's work. Sutton and Barto write that Samuel was the first to propose and implement a learning method built on temporal-difference ideas, in Samuel's checkers program
Dynamic Programming (1957)
Documented direct influence
Sutton and Barto trace one of the roots of reinforcement learning to dynamic programming and the Bellman equation; temporal-difference methods approximate that equation
The conditioned reflex (April 23, 1903)
Inherited, with no proven direct link
Sutton and Barto also develop temporal-difference learning as a model of classical conditioning, the phenomenon Pavlov described
Leads to
TD-Gammon (1992)
Documented direct influence
Tesauro applies the algorithm directly
Q-learning (1989)
Documented direct influence
According to Sutton and Barto, Q-learning brings together the temporal-difference thread and the optimal-control thread in 1989
Turing Award for Barto and Sutton (March 5, 2025)
Documented direct influence
The ACM honors the conceptual and algorithmic foundations of reinforcement learning, temporal-difference learning among them
Threads that pass through here
- From reward to alignment (step 4 of 12)
- Machines versus champions (step 2 of 7)
Sources
Original source
- Learning to Predict by the Methods of Temporal Differences. Machine Learning, 3(1), pp. 9-44 (opens in a new tab). Richard S. Sutton, 1988
- Reinforcement Learning: An Introduction, 1st edition, MIT Press. Section "History of Reinforcement Learning" of the online edition (opens in a new tab), apartado «History of Reinforcement Learning». Richard S. Sutton and Andrew G. Barto, 1998
Scholarship and context
- ACM A.M. Turing Award Honors Two Researchers Who Led the Development of Cornerstone AI Technology (ACM press release, March 5, 2025) (opens in a new tab). Association for Computing Machinery, 2025
How to cite
“Temporal differences.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/sutton-temporal-differences
This entry was last reviewed on November 13, 2026.