DQN
Era 4 · Deep learning and foundation models (2012–2021) › Deep reinforcement learning
- Lineage
- Data and learning
- Kind
- Publication
Who
DeepMind
What happened
Volodymyr Mnih and colleagues at DeepMind post on arXiv a convolutional network that learns to play Atari video games from the screen's pixels and the score, using a variant of Q-learning. The expanded version appears in Nature on February 25, 2015.
Why it mattered
A single program, with nothing changed, learns to play different games using only what it sees on the screen. It opens the field of deep reinforcement learning.
The full story
Q-learning learns how much each action is worth in each situation, but it needs someone to describe the situation with a handful of variables. With a video game screen as input, the number of possible situations is immense, and attempts to approximate the values with neural networks tended to become unstable.
The DeepMind team combined Q-learning with a convolutional network that took the pixels directly. To stabilize training, the program stored its experiences and learned from samples drawn at random from that memory, instead of learning only from whatever had just happened.
In the December 2013 paper they tested it on seven Atari 2600 games without changing the architecture or the algorithm. It beat earlier methods on six of them and an expert human player on three. The Nature version, from February 2015, with nineteen authors, was titled "Human-level control through deep reinforcement learning."
The result showed that a network could learn complex behavior from a reward alone, without rules written for each game. David Silver and Ioannis Antonoglou, two of its authors, would go on to coauthor AlphaGo. Between the two versions of the paper, in 2014, Google had bought DeepMind.
Under the hood
Q-learning estimates Q(s, a), the future reward to be expected from taking action a in situation s. DQN approximates it with a convolutional network and trains the network to bring it closer to r + γ · maxₐ' Q(s', a'): the immediate reward plus the best value of the next situation, discounted by γ.
Two devices keep training from becoming unstable. Experience replay stores transitions in a memory and trains on random samples, which breaks the correlation between consecutive steps. In the Nature version, in addition, the target is computed with a copy of the network that is updated only every so many steps. The input is the last four frames, scaled down to 84 × 84 grayscale pixels.
Videos
- DQN Breakout (opens in a new tab) (Google DeepMind)
Builds on
Q-learning (1989)
Documented direct influence
DQN is Q-learning with a deep network
Leads to
Reinforcement learning from human preferences (June 12, 2017)
Inherited, with no proven direct link
The paper starts from the success of deep reinforcement learning on problems with a well-defined reward (citing Mnih et al., 2015) and carries it to tasks with no reward function, including the same Atari games. For Atari it uses A2C, not DQN
AlphaGo beats Fan Hui (October 5, 2015)
Inherited, with no proven direct link
The same lab and the same combination of reinforcement learning and deep networks; David Silver is an author of both papers. AlphaGo does not use the DQN algorithm
Threads that pass through here
- From reward to alignment (step 7 of 12)
- From Atari to Go (step 1 of 4)
Sources
- Playing Atari with Deep Reinforcement Learning (arXiv:1312.5602) (opens in a new tab). Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller, 2013
- Human-level control through deep reinforcement learning. Nature, 518(7540), pp. 529-533 (opens in a new tab). Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al., 2015
How to cite
“DQN.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/dqn
This entry was last reviewed on November 13, 2026.