The History of AI

PPO

Era 4 · Deep learning and foundation models (2012–2021) › Deep reinforcement learning

Lineage
Data and learning
Kind
Publication

Who

OpenAI

What happened

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov of OpenAI publish proximal policy optimization (PPO), a family of reinforcement learning methods that is simpler to implement than its predecessors and performs as well or better.

Why it mattered

A stable, easy-to-use reinforcement learning algorithm that becomes OpenAI's default method and, years later, the component that fine-tunes language models with human feedback.

The full story

Policy gradient methods improve the agent's strategy directly, but they have a problem: a learning step that is too large can wreck a strategy that was working, and the agent takes a long time to recover. In 2015 TRPO solved this by limiting how much the policy can change in each update, but it was complicated to implement and expensive to compute.

PPO achieves almost the same thing with a simpler device. Its objective function clips the improvement when the new policy strays too far from the old one, so there is no incentive to take large steps. That also makes it possible to reuse each batch of data for several passes of training instead of using it only once.

The authors tested it on simulated robotic locomotion tasks and on Atari games, and according to the abstract it outperformed other policy gradient methods while striking a good balance between sample efficiency, simplicity, and wall-clock time.

From 2018 on, PPO was OpenAI's default reinforcement learning algorithm. It was used to control robotic hands and to beat professional Dota 2 players, and later to fine-tune language models with human feedback, as told in the alignment field.

Leads to

  • InstructGPT (March 4, 2022)

    Documented direct influence

    PPO is the algorithm used to train the policy. The InstructGPT paper fine-tunes the supervised model "using PPO (Schulman et al., 2017)" (section 3.5)

  • RLHF for language models (September 18, 2019)

    Documented direct influence

    The policy is optimized with PPO (Schulman et al., 2017), in its PPO2 version (sections 2 and 2.2)

  • OpenAI Five beats the Dota 2 world champions (April 13, 2019)

    Documented direct influence

    OpenAI Five was trained with PPO, OpenAI's policy-gradient method

Threads that pass through here

Sources

Original source

How to cite

“PPO.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/ppo

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline