InstructGPT
Era 5 · Generative AI, reasoning, and agents (2022–2026) › Alignment and reasoning
- Lineage
- Data and learning
- Kind
- Publication
Who
OpenAI
What happened
Long Ouyang and colleagues at OpenAI describe InstructGPT: GPT-3 fine-tuned first on examples written by people and then with reinforcement learning from their preferences. The outputs of a 1.3-billion-parameter InstructGPT are preferred to those of the 175-billion-parameter GPT-3.
Why it mattered
Reinforcement learning from human feedback applied to GPT-3: the model follows instructions. It is the basis of ChatGPT.
The full story
GPT-3 knew a great deal, but it was not built to do as it was told. Trained to continue internet text, it could respond to a request by making things up, being offensive, or answering something else. The authors summed it up this way: "Making language models bigger does not inherently make them better at following a user's intent."
The method had three steps. First, a group of labelers wrote examples of the desired behavior, and GPT-3 was fine-tuned on them in a supervised way. Next, the labelers ranked several of the model's responses from best to worst, and those rankings trained a reward model. Finally, the language model was fine-tuned with reinforcement learning to maximize that reward.
In human evaluations, outputs from a 1.3-billion-parameter InstructGPT were preferred to those of the original 175-billion-parameter GPT-3, despite having about a hundred times fewer parameters. It also improved in truthfulness and produced less toxic text, with little loss on standard benchmarks, although it still made simple mistakes.
OpenAI had presented the models on January 27, 2022, and the paper appeared on arXiv on March 4. In November of that year the same method, applied to a model in the GPT-3.5 series, gave rise to ChatGPT.
Under the hood
The three phases used separate datasets: about 13,000 prompts with responses written by the labelers for supervised fine-tuning, about 33,000 for training the reward model, and about 31,000 for reinforcement learning.
For the reward model, labelers ranked between 4 and 9 responses to the same prompt, and each ranked pair was used as one example. The model learns to give a higher score to the preferred response: its loss is -log σ(r(x, y_better) - r(x, y_worse)).
In the last phase, the PPO algorithm fine-tunes the model to maximize that score, with a penalty for drifting away from the supervised model. Without it, the model learns to fool the judge with odd responses that score high. The PPO-ptx variant also mixes in pretraining gradients so as not to lose performance on standard benchmarks.
Side by side
| Phase | What people contribute | What is trained |
|---|---|---|
| 1. Supervised fine-tuning | Example responses written by hand | GPT-3 imitates those responses |
| 2. Reward model | Model responses ranked from best to worst | A judge that scores responses |
| 3. Reinforcement learning | Nothing new | The model seeks the judge's highest score without drifting far from step 1 |
Builds on
Reinforcement learning from human preferences (June 12, 2017)
Documented direct influence
The same technique, by way of Ziegler (2019)
PPO (July 20, 2017)
Documented direct influence
PPO is the algorithm used to train the policy. The InstructGPT paper fine-tunes the supervised model "using PPO (Schulman et al., 2017)" (section 3.5)
RLHF for language models (September 18, 2019)
Documented direct influence
"Our methodology follows that of Ziegler et al. (2019) and Stiennon et al. (2020)" (section 3.1)
GPT-3 (May 28, 2020)
Documented direct influence
InstructGPT is GPT-3 fine-tuned first on demonstrations written by people and then with reinforcement learning from their preferences (abstract)
Leads to
ChatGPT (November 30, 2022)
Documented direct influence
OpenAI presents ChatGPT as a sibling model to InstructGPT, trained with RLHF using the same methods
Constitutional AI (December 15, 2022)
Inherited, with no proven direct link
Constitutional AI keeps the scheme of reinforcement learning with a preference model and replaces human labels on harmful responses with labels from an AI guided by principles
DPO (May 29, 2023)
Documented direct influence
DPO takes as its starting point the RLHF procedure of Ziegler, Stiennon, and Ouyang (InstructGPT), with a reward model and PPO, and replaces it with a single classification loss (sections 1 and 3)
Threads that pass through here
- From reward to alignment (step 11 of 12)
Sources
Original source
Scholarship and context
- Training language models to follow instructions with human feedback. NeurIPS 2022 (opens in a new tab). Long Ouyang et al., 2022
How to cite
“InstructGPT.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/instructgpt
This entry was last reviewed on November 13, 2026.