Learn · Module 11 · Evaluation · Lesson 3 of 5
AI judges
A language model can grade outputs at scale, but only after you've checked that it agrees with people.
In short
- AI judges grade outputs at scale using a rubric, but must be validated against people.
- Measure agreement beyond chance, for example with Cohen's kappa, and read the disagreements.
- Watch for position, length and self-preference bias.
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.