RLHF for language models
Era 4 · Deep learning and foundation models (2012–2021) › Alignment
- Lineage
- Data and learning
- Kind
- Application
Who
OpenAI
What happened
Daniel Ziegler and colleagues at OpenAI fine-tune pretrained language models with a reward model learned from human judgments, on tasks of continuing text in a given style and of summarization.
Why it mattered
Learning from human preferences reaches language models. It is the technique that, reused in InstructGPT, will turn a language model into an assistant.
The full story
A pretrained language model knows how to continue text, but it does not know what text its user wants. For tasks such as writing in a positive tone or summarizing well, there is no formula that measures success. Ziegler and colleagues carried Christiano's method over to this setting: people compare the model's responses, a reward model learns their preferences, and the language model is fine-tuned with reinforcement learning to satisfy it.
They tested it on four tasks. On two text continuation tasks, one with a positive tone and one with physical descriptions, 5,000 human comparisons were enough for good results. On two summarization tasks, with forum posts and with news stories from CNN and the Daily Mail, they used 60,000.
The summaries were rated well by the evaluators, but the authors flagged a problem: the models tended to copy whole sentences from the original text, skipping the irrelevant opening, and may have been exploiting simple shortcuts the raters relied on. It is an early form of a known risk of the method: the model learns what gets rated well, which is not always what is good.
The algorithm used today comes from this work, and it became widespread when it was reused in InstructGPT, the predecessor of ChatGPT.
Builds on
Reinforcement learning from human preferences (June 12, 2017)
Documented direct influence
The paper has labelers choose among several options, following Christiano et al. (2017) (section 2)
PPO (July 20, 2017)
Documented direct influence
The policy is optimized with PPO (Schulman et al., 2017), in its PPO2 version (sections 2 and 2.2)
Leads to
InstructGPT (March 4, 2022)
Documented direct influence
"Our methodology follows that of Ziegler et al. (2019) and Stiennon et al. (2020)" (section 3.1)
Threads that pass through here
- From reward to alignment (step 10 of 12)
Sources
Original source
- Fine-Tuning Language Models from Human Preferences (arXiv:1909.08593) (opens in a new tab). Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving, 2019
How to cite
“RLHF for language models.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/rlhf-for-language-models
This entry was last reviewed on November 13, 2026.