Reinforcement learning from human preferences
Era 4 · Deep learning and foundation models (2012–2021) › Alignment
- Lineage
- Data and learning
- Kind
- Publication
Who
OpenAI, DeepMind
What happened
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei, of OpenAI and DeepMind, train reinforcement learning agents with no reward function, using only comparisons from people who choose which of two clips of behavior they prefer.
Why it mattered
The origin of reinforcement learning from human feedback (RLHF): a model learns what people want from their comparisons, and the agent is trained to satisfy that model.
The full story
Reinforcement learning needs a reward: a number that says whether the agent is doing well. In a video game the score provides it, but for many real tasks nobody knows how to write one, and a badly specified reward produces an agent that maximizes it in ways nobody wanted.
Christiano and colleagues proposed that the system learn the reward itself, from people. The agent acts, the system shows a person two short clips of its behavior, and the person picks the one closer to what they are looking for. Those comparisons train a reward model, and the agent learns to maximize what that model predicts. The raters did not need to be experts.
The method worked on Atari games and on simulated robotic locomotion tasks with human comparisons for less than 1 percent of the agent's interactions with its environment, few enough to make it practical. With about an hour of human time, the authors taught new behaviors that would have been hard to describe with a written reward.
The paper was written by researchers from OpenAI and DeepMind, and several of them kept working along this line. Two years later the same method would be applied to language models.
Four boxes in a circle. "Agent" leads to "two clips of its behavior," then to "a person picks the better one," then to "reward model" with the label "comparisons," and a colored arrow returns to the agent with the label "learned reward, which the agent maximizes."
Under the hood
The reward model r̂ is fitted so that it explains the human choices. The probability that the person prefers clip σ¹ to clip σ² is modeled as exp Σ r̂(σ¹) / (exp Σ r̂(σ¹) + exp Σ r̂(σ²)), where each sum runs over the steps of the clip, and it is trained by minimizing the cross-entropy with the actual choices.
The agent and the reward model are trained at the same time: the agent generates new clips, people compare some of them, and the model is updated. To choose which pairs to show, the authors favored those on which several reward models trained in parallel disagreed most.
Common misconceptions
What people often say: Learning from human feedback was born with ChatGPT.
In fact: The method dates from 2017 and was tested on Atari games and simulated robots. It reached language models in 2019 and ChatGPT by way of InstructGPT.
Builds on
DQN (December 19, 2013)
Inherited, with no proven direct link
The paper starts from the success of deep reinforcement learning on problems with a well-defined reward (citing Mnih et al., 2015) and carries it to tasks with no reward function, including the same Atari games. For Atari it uses A2C, not DQN
Leads to
InstructGPT (March 4, 2022), ChatGPT (November 30, 2022)
Documented direct influence
The same technique, by way of Ziegler (2019)
RLHF for language models (September 18, 2019)
Documented direct influence
The paper has labelers choose among several options, following Christiano et al. (2017) (section 2)
Anthropic is founded (January 2021)
Documented direct influence
The announcement of Anthropic's first funding round presents its team as the authors of earlier research on GPT-3, scaling laws, and learning from human preferences
Threads that pass through here
- From reward to alignment (step 8 of 12)
- Who pays for compute (step 2 of 6)
Sources
Original source
- Deep reinforcement learning from human preferences (arXiv:1706.03741) (opens in a new tab). Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei, 2017
How to cite
“Reinforcement learning from human preferences.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/rl-from-human-preferences
This entry was last reviewed on November 13, 2026.