It's hard to write the perfect answer, but easy to say which of two answers is better. Preference tuning learns from exactly that.
4 minute read
In short
Preference data records which of two answers people prefer.
RLHF trains a reward model then optimises against it; DPO trains on the pairs directly.
Watch for reward hacking: optimising the score instead of real quality.
In plain words
After supervised fine-tuning, a model can follow instructions. To make it more helpful, honest and safe, labs collect preferences: show people two answers to the same prompt, and record which one they prefer.
RLHF (reinforcement learning from human feedback) first trains a reward model to predict which answer people prefer, then trains the chat model to produce answers the reward model scores highly. DPO (direct preference optimisation) skips the reward model and trains on the preference pairs directly.
Worked example: from scores to a preference
A reward model gives answer A a score of 2.0 and answer B a score of 0.5. The chance it predicts people prefer A is sigmoid(score A − score B).
1
Difference in scores
2.0 − 0.5 = 1.5
2
Turn it into a probability
sigmoid(1.5) = 0.818
3
Learn from the label
If people really did prefer A, the reward model is fairly right and gets a small update. If they preferred B (probability 0.182 by its own numbers), it gets a large one.
Worked example: DPO's loss
DPO looks at how much more likely the model makes the preferred answer, compared with the rejected one, relative to where it started. Call that the margin. The loss is −ln(sigmoid(β × margin)), with β often around 0.1.
margin
sigmoid(0.1 × margin)
loss
+5 (prefers the good answer)
0.622
0.474
0 (no preference yet)
0.500
0.693
−5 (prefers the bad answer)
0.378
0.974
Training pushes the margin up: the model learns to make preferred answers more likely and rejected ones less likely.
Your turn
Why collect 'A is better than B' instead of asking people to write perfect answers?
What to remember
Preference tuning shapes behaviour: helpfulness, tone, refusals. It can also go wrong. A model can learn to please the reward model rather than people, for example by being longer or more flattering than needed. This is called reward hacking, and checking real outputs, not just scores, is how you catch it.
Common mistakes
Describing RLHF as the model learning facts from humans; it shapes style and preferences, not knowledge.
Ignoring reward hacking, where the model learns to please the reward model rather than the user.
Assuming RLHF is the only option. DPO trains on preferences directly without a separate reward model.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
Explain RLHF at a high level.
RLHF, reinforcement learning from human feedback, makes a model's answers more helpful and safer. After supervised fine-tuning, people compare pairs of model answers to the same prompt and pick the better one. Those comparisons train a reward model that predicts which answer humans would prefer. Then the language model is trained with reinforcement learning, often PPO, to produce answers that score highly with the reward model, with a penalty for drifting too far from the original model so it doesn't learn to game the reward. A simpler alternative, DPO, skips the reward model and trains directly on the preference pairs. Both shape behaviour like helpfulness, tone and refusals rather than adding knowledge.