DPO
Era 5 · Generative AI, reasoning, and agents (2022–2026) › Alignment and reasoning
- Lineage
- Data and learning
- Kind
- Publication
Who
Stanford University
What happened
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn of Stanford propose direct preference optimization (DPO): aligning a language model with human preferences through a simple loss function, with no separate reward model and no reinforcement learning.
Why it mattered
Preference alignment with no reward model and no reinforcement learning loop, simpler to implement and to train.
The full story
Learning from human feedback had two delicate phases. First a reward model had to be trained on people's preferences. Then the language model had to be fine-tuned with reinforcement learning to maximize that reward without drifting too far from the original. The authors described it as a complex and often unstable procedure.
Their contribution was mathematical. They showed that, with the right way of writing the reward model, the optimal policy can be obtained directly, and that the problem reduces to a simple classification loss: raise the probability of the preferred response relative to the rejected one in each pair. The language model itself serves as the reward model, as the paper's subtitle says: "Your Language Model is Secretly a Reward Model."
DPO does not need to sample from the model during training and requires almost no hyperparameter tuning. In the experiments it matched or beat existing methods: it controlled the sentiment of generated text better than reinforcement learning with PPO and matched or improved quality in summarization and dialogue, while being easier to implement and to train.
The paper was presented at NeurIPS 2023. Its simplicity made it an attractive alternative to reinforcement learning for fine-tuning models on preferences.
Three rows. RLHF and constitutional AI have three steps (fine-tuning, judge, and reinforcement with PPO); in the first, the judge learns from people who rank responses, and in the second, from an AI that compares them. The DPO row is a single box, a loss over pairs of responses with no reward model and no reinforcement.
Under the hood
With RLHF, four models have to be kept during training: the one being fine-tuned, a frozen copy of the original, the reward model, and the one that estimates value in PPO. DPO needs only the first two.
For each prompt x with a preferred response y_w and a rejected one y_l, DPO minimizes -log σ(β·log[π(y_w|x)/π_ref(y_w|x)] - β·log[π(y_l|x)/π_ref(y_l|x)]), where π is the model being fine-tuned and π_ref the frozen copy.
In words: raise the probability of the preferred response and lower that of the rejected one, always measured against the original model. The parameter β controls how far the model may drift from it, the same role played by the penalty in RLHF. The implicit reward of a response is β·log[π(y|x)/π_ref(y|x)], hence the paper's subtitle.
Side by side
| Aspect | RLHF | DPO |
|---|---|---|
| Separate reward model | Yes | No: the model itself plays that role |
| Reinforcement learning | Yes (PPO) | No |
| Sampling from the model during training | Yes | No |
| What is optimized | The reward model's score | A classification loss over pairs |
Builds on
InstructGPT (March 4, 2022)
Documented direct influence
DPO takes as its starting point the RLHF procedure of Ziegler, Stiennon, and Ouyang (InstructGPT), with a reward model and PPO, and replaces it with a single classification loss (sections 1 and 3)
Sources
Original source
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv:2305.18290) (opens in a new tab). Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn, 2023. open license: CC BY 4.0
Scholarship and context
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 (opens in a new tab). Rafael Rafailov et al., 2023
How to cite
“DPO.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/dpo
This entry was last reviewed on November 13, 2026.