epochs

Learn · Module 07 · Training and fine-tuning · Lesson 6 of 8

Learning from preferences

It's hard to write the perfect answer, but easy to say which of two answers is better. Preference tuning learns from exactly that.

4 minute read

In short
  • Preference data records which of two answers people prefer.
  • RLHF trains a reward model then optimises against it; DPO trains on the pairs directly.
  • Watch for reward hacking: optimising the score instead of real quality.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.