epochs

Learn · Module 11 · Evaluation · Lesson 1 of 5

Deciding what good means

You can't improve a system until you can say, in checkable terms, what a good output is.

3 minute read

In short
  • Turn vague goals into specific, checkable criteria.
  • Use code checks where possible and AI or human judges for meaning.
  • Set a target for each criterion before you start building.

In plain words

'Make the summaries better' can't be measured. Before evaluating anything, turn the goal into success criteria: specific, checkable statements about a good output. Each one should be something two people would agree on when looking at the same output.

Worked example: a meeting-notes summariser

vague goalcheckable criterionhow to check
AccurateEvery decision and owner named matches the transcriptAI judge with the transcript, spot-checked by people
CompleteIncludes all action items (we have a labelled list for 60 meetings)Code: compare against the list
ShortUnder 150 wordsCode: count words
Useful formatHas sections Decisions, Actions, Open questionsCode: check headings
No made-up contentNo names or dates absent from the transcriptAI judge, plus people on a sample
  1. 1
    Prefer code checks

    Length, format and exact matches are cheap, fast and never disagree with themselves.

  2. 2
    Use judges for meaning

    Accuracy and tone need an AI judge or a person, so write a clear rubric for each.

  3. 3
    Set a target for each

    For example: 95% have every action item, 100% under 150 words, under 2% contain anything invented.

Your turn

Which of these is a usable success criterion?

What to remember

Write criteria before building, from what users actually need. Keep the list short. Collect examples of good and bad outputs for each criterion; they become your first test set and the examples for any AI judge.

Common mistakes

  • Defining success as 'the answers look good' without anything measurable.
  • Picking one metric and ignoring guardrails like cost, latency or safety.
  • Setting targets without a baseline, so you don't know what's realistic.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

How would you define success for an LLM feature?

I'd start from the user and business goal, then make it measurable. For a support assistant, success might be: correct answers on at least 90% of a test set built from real questions, no answers that contradict policy, a hand-off to a human whenever the user asks or the case is sensitive, a response in under three seconds, and a cost under a set amount per conversation. Each criterion needs a way to measure it, like exact checks, an AI judge with a rubric, or human review, and a threshold agreed before building. Online, I'd add metrics like resolution rate and satisfaction. Writing these down first stops endless debates about whether it's 'good enough'.

More to think about
  • How do you evaluate output that has no single right answer?
  • What's a good first step when asked to evaluate an existing AI product?

Go further

Was this lesson clear?