The History of AI

RLHF for language models

Era 4 · Deep learning and foundation models (2012–2021) › Alignment

Lineage
Data and learning
Kind
Application

Who

OpenAI

What happened

Daniel Ziegler and colleagues at OpenAI fine-tune pretrained language models with a reward model learned from human judgments, on tasks of continuing text in a given style and of summarization.

Why it mattered

Learning from human preferences reaches language models. It is the technique that, reused in InstructGPT, will turn a language model into an assistant.

The full story

A pretrained language model knows how to continue text, but it does not know what text its user wants. For tasks such as writing in a positive tone or summarizing well, there is no formula that measures success. Ziegler and colleagues carried Christiano's method over to this setting: people compare the model's responses, a reward model learns their preferences, and the language model is fine-tuned with reinforcement learning to satisfy it.

They tested it on four tasks. On two text continuation tasks, one with a positive tone and one with physical descriptions, 5,000 human comparisons were enough for good results. On two summarization tasks, with forum posts and with news stories from CNN and the Daily Mail, they used 60,000.

The summaries were rated well by the evaluators, but the authors flagged a problem: the models tended to copy whole sentences from the original text, skipping the irrelevant opening, and may have been exploiting simple shortcuts the raters relied on. It is an early form of a known risk of the method: the model learns what gets rated well, which is not always what is good.

The algorithm used today comes from this work, and it became widespread when it was reused in InstructGPT, the predecessor of ChatGPT.

Builds on

  • Reinforcement learning from human preferences (June 12, 2017)

    Documented direct influence

    The paper has labelers choose among several options, following Christiano et al. (2017) (section 2)

  • PPO (July 20, 2017)

    Documented direct influence

    The policy is optimized with PPO (Schulman et al., 2017), in its PPO2 version (sections 2 and 2.2)

Leads to

  • InstructGPT (March 4, 2022)

    Documented direct influence

    "Our methodology follows that of Ziegler et al. (2019) and Stiennon et al. (2020)" (section 3.1)

Threads that pass through here

Sources

Original source

How to cite

“RLHF for language models.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/rlhf-for-language-models

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline