DPO: The RLHF Alternative That Trains LLMs Without a Reward Model
Direct Preference Optimization eliminates the separate reward model and PPO loop from RLHF, deriving an equivalent alignment objective directly from preference data with a simple classification loss.
Standard RLHF requires training and running three separate models simultaneously: the SFT (supervised fine-tuned) policy, the reward model, and the RLHF-updated policy (kept close to SFT via KL penalty). PPO introduces additional hyperparameters (clip ratio, value function, GAE lambda) and training instability. The entire system is difficult to debug and expensive to run.
The DPO Insight
The DPO paper (arXiv:2305.18290) by Rafailov et al. shows that the optimal RLHF policy has a closed-form solution in terms of the reference policy (SFT model) and the reward function. Substituting this back into the reward modeling objective produces a loss that depends only on the policy being trained - no separate reward model needed.
where y_w is the preferred response, y_l is the rejected response, pi is the model being trained, pi_ref is the frozen SFT reference, and beta controls how closely the model should stay near the reference.
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
DPO increases the relative log-probability of preferred responses over rejected responses, weighted by how much the current policy already agrees with the reference. Responses where the model has already diverged from the reference are downweighted, preventing overtraining on easy examples.
Preference Dataset Format
DPO requires a dataset of (prompt, chosen, rejected) triplets - the same format as RLHF reward model training. You can use HuggingFace's Anthropic HH-RLHF dataset, Ultrafeedback, or Preference 700K directly.
DPO training is simpler, faster, and more stable than PPO-based RLHF. However, it lacks the iterative online learning of RLHF - the preference data is fixed. Online DPO and SPIN (self-play fine-tuning) variants attempt to address this by generating new samples during training.
ORPO and SimPO Variants
ORPO (Odds Ratio Preference Optimization) eliminates the need for a reference model entirely by folding the alignment signal into the SFT loss. SimPO (Simple Preference Optimization) removes the reference model while using a length-normalized reward, preventing verbosity bias.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Transfer learning lets you start from a pretrained model instead of random weights. Here is why it works, when to fine-tune vs. freeze layers, and when it fails.
