The History of AI

DPO

Era 5 · Generative AI, reasoning, and agents (2022–2026) › Alignment and reasoning

Lineage
Data and learning
Kind
Publication

Who

Stanford University

What happened

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn of Stanford propose direct preference optimization (DPO): aligning a language model with human preferences through a simple loss function, with no separate reward model and no reinforcement learning.

Why it mattered

Preference alignment with no reward model and no reinforcement learning loop, simpler to implement and to train.

The full story

Learning from human feedback had two delicate phases. First a reward model had to be trained on people's preferences. Then the language model had to be fine-tuned with reinforcement learning to maximize that reward without drifting too far from the original. The authors described it as a complex and often unstable procedure.

Their contribution was mathematical. They showed that, with the right way of writing the reward model, the optimal policy can be obtained directly, and that the problem reduces to a simple classification loss: raise the probability of the preferred response relative to the rejected one in each pair. The language model itself serves as the reward model, as the paper's subtitle says: "Your Language Model is Secretly a Reward Model."

DPO does not need to sample from the model during training and requires almost no hyperparameter tuning. In the experiments it matched or beat existing methods: it controlled the sentiment of generated text better than reinforcement learning with PPO and matched or improved quality in summarization and dialogue, while being easier to implement and to train.

The paper was presented at NeurIPS 2023. Its simplicity made it an attractive alternative to reinforcement learning for fine-tuning models on preferences.

Three rows. RLHF and constitutional AI have three steps (fine-tuning, judge, and reinforcement with PPO); in the first, the judge learns from people who rank responses, and in the second, from an AI that compares them. The DPO row is a single box, a loss over pairs of responses with no reward model and no reinforcement.

RLHF and constitutional AI train a judge and optimize against it; DPO reaches the same goal with a single loss.

Under the hood

With RLHF, four models have to be kept during training: the one being fine-tuned, a frozen copy of the original, the reward model, and the one that estimates value in PPO. DPO needs only the first two.

For each prompt x with a preferred response y_w and a rejected one y_l, DPO minimizes -log σ(β·log[π(y_w|x)/π_ref(y_w|x)] - β·log[π(y_l|x)/π_ref(y_l|x)]), where π is the model being fine-tuned and π_ref the frozen copy.

In words: raise the probability of the preferred response and lower that of the rejected one, always measured against the original model. The parameter β controls how far the model may drift from it, the same role played by the penalty in RLHF. The implicit reward of a response is β·log[π(y|x)/π_ref(y|x)], hence the paper's subtitle.

Side by side

RLHF and DPO
AspectRLHFDPO
Separate reward modelYesNo: the model itself plays that role
Reinforcement learningYes (PPO)No
Sampling from the model during trainingYesNo
What is optimizedThe reward model's scoreA classification loss over pairs

Builds on

  • InstructGPT (March 4, 2022)

    Documented direct influence

    DPO takes as its starting point the RLHF procedure of Ziegler, Stiennon, and Ouyang (InstructGPT), with a reward model and PPO, and replaces it with a single classification loss (sections 1 and 3)

Sources

Original source

How to cite

“DPO.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/dpo

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline