The History of AI

InstructGPT

Era 5 · Generative AI, reasoning, and agents (2022–2026) › Alignment and reasoning

Lineage
Data and learning
Kind
Publication

Who

OpenAI

What happened

Long Ouyang and colleagues at OpenAI describe InstructGPT: GPT-3 fine-tuned first on examples written by people and then with reinforcement learning from their preferences. The outputs of a 1.3-billion-parameter InstructGPT are preferred to those of the 175-billion-parameter GPT-3.

Why it mattered

Reinforcement learning from human feedback applied to GPT-3: the model follows instructions. It is the basis of ChatGPT.

The full story

GPT-3 knew a great deal, but it was not built to do as it was told. Trained to continue internet text, it could respond to a request by making things up, being offensive, or answering something else. The authors summed it up this way: "Making language models bigger does not inherently make them better at following a user's intent."

The method had three steps. First, a group of labelers wrote examples of the desired behavior, and GPT-3 was fine-tuned on them in a supervised way. Next, the labelers ranked several of the model's responses from best to worst, and those rankings trained a reward model. Finally, the language model was fine-tuned with reinforcement learning to maximize that reward.

In human evaluations, outputs from a 1.3-billion-parameter InstructGPT were preferred to those of the original 175-billion-parameter GPT-3, despite having about a hundred times fewer parameters. It also improved in truthfulness and produced less toxic text, with little loss on standard benchmarks, although it still made simple mistakes.

OpenAI had presented the models on January 27, 2022, and the paper appeared on arXiv on March 4. In November of that year the same method, applied to a model in the GPT-3.5 series, gave rise to ChatGPT.

Under the hood

The three phases used separate datasets: about 13,000 prompts with responses written by the labelers for supervised fine-tuning, about 33,000 for training the reward model, and about 31,000 for reinforcement learning.

For the reward model, labelers ranked between 4 and 9 responses to the same prompt, and each ranked pair was used as one example. The model learns to give a higher score to the preferred response: its loss is -log σ(r(x, y_better) - r(x, y_worse)).

In the last phase, the PPO algorithm fine-tunes the model to maximize that score, with a penalty for drifting away from the supervised model. Without it, the model learns to fool the judge with odd responses that score high. The PPO-ptx variant also mixes in pretraining gradients so as not to lose performance on standard benchmarks.

Side by side

The three phases of InstructGPT
PhaseWhat people contributeWhat is trained
1. Supervised fine-tuningExample responses written by handGPT-3 imitates those responses
2. Reward modelModel responses ranked from best to worstA judge that scores responses
3. Reinforcement learningNothing newThe model seeks the judge's highest score without drifting far from step 1

Builds on

  • Reinforcement learning from human preferences (June 12, 2017)

    Documented direct influence

    The same technique, by way of Ziegler (2019)

  • PPO (July 20, 2017)

    Documented direct influence

    PPO is the algorithm used to train the policy. The InstructGPT paper fine-tunes the supervised model "using PPO (Schulman et al., 2017)" (section 3.5)

  • RLHF for language models (September 18, 2019)

    Documented direct influence

    "Our methodology follows that of Ziegler et al. (2019) and Stiennon et al. (2020)" (section 3.1)

  • GPT-3 (May 28, 2020)

    Documented direct influence

    InstructGPT is GPT-3 fine-tuned first on demonstrations written by people and then with reinforcement learning from their preferences (abstract)

Leads to

  • ChatGPT (November 30, 2022)

    Documented direct influence

    OpenAI presents ChatGPT as a sibling model to InstructGPT, trained with RLHF using the same methods

  • Constitutional AI (December 15, 2022)

    Inherited, with no proven direct link

    Constitutional AI keeps the scheme of reinforcement learning with a preference model and replaces human labels on harmful responses with labels from an AI guided by principles

  • DPO (May 29, 2023)

    Documented direct influence

    DPO takes as its starting point the RLHF procedure of Ziegler, Stiennon, and Ouyang (InstructGPT), with a reward model and PPO, and replaces it with a single classification loss (sections 1 and 3)

Threads that pass through here

Sources

How to cite

“InstructGPT.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/instructgpt

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline