The History of AI

DQN

Era 4 · Deep learning and foundation models (2012–2021) › Deep reinforcement learning

Lineage
Data and learning
Kind
Publication

Who

DeepMind

What happened

Volodymyr Mnih and colleagues at DeepMind post on arXiv a convolutional network that learns to play Atari video games from the screen's pixels and the score, using a variant of Q-learning. The expanded version appears in Nature on February 25, 2015.

Why it mattered

A single program, with nothing changed, learns to play different games using only what it sees on the screen. It opens the field of deep reinforcement learning.

The full story

Q-learning learns how much each action is worth in each situation, but it needs someone to describe the situation with a handful of variables. With a video game screen as input, the number of possible situations is immense, and attempts to approximate the values with neural networks tended to become unstable.

The DeepMind team combined Q-learning with a convolutional network that took the pixels directly. To stabilize training, the program stored its experiences and learned from samples drawn at random from that memory, instead of learning only from whatever had just happened.

In the December 2013 paper they tested it on seven Atari 2600 games without changing the architecture or the algorithm. It beat earlier methods on six of them and an expert human player on three. The Nature version, from February 2015, with nineteen authors, was titled "Human-level control through deep reinforcement learning."

The result showed that a network could learn complex behavior from a reward alone, without rules written for each game. David Silver and Ioannis Antonoglou, two of its authors, would go on to coauthor AlphaGo. Between the two versions of the paper, in 2014, Google had bought DeepMind.

Under the hood

Q-learning estimates Q(s, a), the future reward to be expected from taking action a in situation s. DQN approximates it with a convolutional network and trains the network to bring it closer to r + γ · maxₐ' Q(s', a'): the immediate reward plus the best value of the next situation, discounted by γ.

Two devices keep training from becoming unstable. Experience replay stores transitions in a memory and trains on random samples, which breaks the correlation between consecutive steps. In the Nature version, in addition, the target is computed with a copy of the network that is updated only every so many steps. The input is the last four frames, scaled down to 84 × 84 grayscale pixels.

Videos

Builds on

  • Q-learning (1989)

    Documented direct influence

    DQN is Q-learning with a deep network

Leads to

  • Reinforcement learning from human preferences (June 12, 2017)

    Inherited, with no proven direct link

    The paper starts from the success of deep reinforcement learning on problems with a well-defined reward (citing Mnih et al., 2015) and carries it to tasks with no reward function, including the same Atari games. For Atari it uses A2C, not DQN

  • AlphaGo beats Fan Hui (October 5, 2015)

    Inherited, with no proven direct link

    The same lab and the same combination of reinforcement learning and deep networks; David Silver is an author of both papers. AlphaGo does not use the DQN algorithm

Threads that pass through here

Sources

How to cite

“DQN.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/dqn

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline