Direct Preference Optimization: Your Language Model Is Secretly a Reward Model (DPO)

Derived an exact closed-form substitution of the reward function in the RLHF objective, training aligned language models with simple binary cross-entropy.

Widely replaced PPO in open-source LLM post-training (Llama-3, Mistral, Zephyr) due to 3x lower VRAM overhead and extreme training stability.

Assumptions

• The implicit reward can be formulated analytically as r(x, y) = beta * log(pi(y|x) / pi_ref(y|x)), eliminating the need for a separate reward model

Limitations

• Sensitive to preference dataset noise; vulnerable to length-exploitation bias without explicit regularization