PPO
Era 4 · Deep learning and foundation models (2012–2021) › Deep reinforcement learning
- Lineage
- Data and learning
- Kind
- Publication
Who
OpenAI
What happened
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov of OpenAI publish proximal policy optimization (PPO), a family of reinforcement learning methods that is simpler to implement than its predecessors and performs as well or better.
Why it mattered
A stable, easy-to-use reinforcement learning algorithm that becomes OpenAI's default method and, years later, the component that fine-tunes language models with human feedback.
The full story
Policy gradient methods improve the agent's strategy directly, but they have a problem: a learning step that is too large can wreck a strategy that was working, and the agent takes a long time to recover. In 2015 TRPO solved this by limiting how much the policy can change in each update, but it was complicated to implement and expensive to compute.
PPO achieves almost the same thing with a simpler device. Its objective function clips the improvement when the new policy strays too far from the old one, so there is no incentive to take large steps. That also makes it possible to reuse each batch of data for several passes of training instead of using it only once.
The authors tested it on simulated robotic locomotion tasks and on Atari games, and according to the abstract it outperformed other policy gradient methods while striking a good balance between sample efficiency, simplicity, and wall-clock time.
From 2018 on, PPO was OpenAI's default reinforcement learning algorithm. It was used to control robotic hands and to beat professional Dota 2 players, and later to fine-tune language models with human feedback, as told in the alignment field.
Leads to
InstructGPT (March 4, 2022)
Documented direct influence
PPO is the algorithm used to train the policy. The InstructGPT paper fine-tunes the supervised model "using PPO (Schulman et al., 2017)" (section 3.5)
RLHF for language models (September 18, 2019)
Documented direct influence
The policy is optimized with PPO (Schulman et al., 2017), in its PPO2 version (sections 2 and 2.2)
OpenAI Five beats the Dota 2 world champions (April 13, 2019)
Documented direct influence
OpenAI Five was trained with PPO, OpenAI's policy-gradient method
Threads that pass through here
- From reward to alignment (step 9 of 12)
Sources
Original source
- Proximal Policy Optimization Algorithms (arXiv:1707.06347) (opens in a new tab). John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, 2017
How to cite
“PPO.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/ppo
This entry was last reviewed on November 13, 2026.