epochs

Learn · Module 11 · Evaluation · Lesson 3 of 5

AI judges

A language model can grade outputs at scale, but only after you've checked that it agrees with people.

4 minute read

In short
  • AI judges grade outputs at scale using a rubric, but must be validated against people.
  • Measure agreement beyond chance, for example with Cohen's kappa, and read the disagreements.
  • Watch for position, length and self-preference bias.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.