The History of AI

Reinforcement learning from human preferences

Era 4 · Deep learning and foundation models (2012–2021) › Alignment

Lineage
Data and learning
Kind
Publication

Who

OpenAI, DeepMind

What happened

Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei, of OpenAI and DeepMind, train reinforcement learning agents with no reward function, using only comparisons from people who choose which of two clips of behavior they prefer.

Why it mattered

The origin of reinforcement learning from human feedback (RLHF): a model learns what people want from their comparisons, and the agent is trained to satisfy that model.

The full story

Reinforcement learning needs a reward: a number that says whether the agent is doing well. In a video game the score provides it, but for many real tasks nobody knows how to write one, and a badly specified reward produces an agent that maximizes it in ways nobody wanted.

Christiano and colleagues proposed that the system learn the reward itself, from people. The agent acts, the system shows a person two short clips of its behavior, and the person picks the one closer to what they are looking for. Those comparisons train a reward model, and the agent learns to maximize what that model predicts. The raters did not need to be experts.

The method worked on Atari games and on simulated robotic locomotion tasks with human comparisons for less than 1 percent of the agent's interactions with its environment, few enough to make it practical. With about an hour of human time, the authors taught new behaviors that would have been hard to describe with a written reward.

The paper was written by researchers from OpenAI and DeepMind, and several of them kept working along this line. Two years later the same method would be applied to language models.

Four boxes in a circle. "Agent" leads to "two clips of its behavior," then to "a person picks the better one," then to "reward model" with the label "comparisons," and a colored arrow returns to the agent with the label "learned reward, which the agent maximizes."

The agent acts, a person compares two clips of its behavior, a reward model learns from those comparisons, and the agent is trained to maximize what that model predicts.

Under the hood

The reward model r̂ is fitted so that it explains the human choices. The probability that the person prefers clip σ¹ to clip σ² is modeled as exp Σ r̂(σ¹) / (exp Σ r̂(σ¹) + exp Σ r̂(σ²)), where each sum runs over the steps of the clip, and it is trained by minimizing the cross-entropy with the actual choices.

The agent and the reward model are trained at the same time: the agent generates new clips, people compare some of them, and the model is updated. To choose which pairs to show, the authors favored those on which several reward models trained in parallel disagreed most.

Common misconceptions

  • What people often say: Learning from human feedback was born with ChatGPT.

    In fact: The method dates from 2017 and was tested on Atari games and simulated robots. It reached language models in 2019 and ChatGPT by way of InstructGPT.

Builds on

  • DQN (December 19, 2013)

    Inherited, with no proven direct link

    The paper starts from the success of deep reinforcement learning on problems with a well-defined reward (citing Mnih et al., 2015) and carries it to tasks with no reward function, including the same Atari games. For Atari it uses A2C, not DQN

Leads to

  • InstructGPT (March 4, 2022), ChatGPT (November 30, 2022)

    Documented direct influence

    The same technique, by way of Ziegler (2019)

  • RLHF for language models (September 18, 2019)

    Documented direct influence

    The paper has labelers choose among several options, following Christiano et al. (2017) (section 2)

  • Anthropic is founded (January 2021)

    Documented direct influence

    The announcement of Anthropic's first funding round presents its team as the authors of earlier research on GPT-3, scaling laws, and learning from human preferences

Threads that pass through here

Sources

Original source

How to cite

“Reinforcement learning from human preferences.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/rl-from-human-preferences

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline