epochs

Learn · Module 11 · Evaluation · Lesson 2 of 5

Scoring against references

When you know the right answer, compare against it. The question is how strict to be.

3 minute read

In short
  • Exact match is right for structured answers like IDs, labels and numbers.
  • Token F1 gives partial credit for overlapping words in short answers.
  • For long free text, word overlap is a poor guide; use a judge with the reference.

In plain words

A reference is a known correct answer. Exact match gives 1 only if the output equals it after simple cleanup (lowercase, trim spaces). Token F1 gives partial credit for overlapping words, combining precision (how much of the output is in the reference) and recall (how much of the reference is in the output).

Worked example: who led the Salt March?

The reference is 'Mahatma Gandhi'.

model's answerexact matchprecisionrecallF1
Mahatma Gandhi11.001.001.000
Gandhi01.000.500.667
Mahatma Gandhi led the movement00.401.000.571
it was Jawaharlal Nehru0000
  1. 1
    'Gandhi' is right but scores 0 on exact match

    Exact match is too strict for free text.

  2. 2
    F1 gives fair partial credit

    'Gandhi' matches 1 of 1 output words (precision 1.0) and 1 of 2 reference words (recall 0.5).

    F1 = 2 × 1.0 × 0.5 ÷ 1.5 = 0.667
  3. 3
    But F1 punishes extra words

    A correct, longer sentence scores lower than the bare name.

Your turn

You're grading a model that extracts invoice numbers like 'INV-20931'. Which scoring is right?

What to remember

Use exact match for structured answers: IDs, labels, numbers, JSON fields. Use F1 or similar overlap scores for short free-text answers. For longer, open-ended answers, overlap scores stop meaning much, and you'll need an AI judge with the reference, the next lesson.

Common mistakes

  • Using exact match for free-text answers, which marks correct paraphrases as wrong.
  • Trusting overlap scores like BLEU or ROUGE for open-ended answers; they reward similar words, not correctness.
  • Writing reference answers so narrow that other valid answers fail.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

How would you score answers when you have a reference answer?

It depends on the kind of answer. For short factual or structured answers, like a category, a number or an ID, I'd use exact match after normalising case, spacing and formats. For extracted fields, I'd score each field separately so I can see which ones fail. For free-text answers, word-overlap scores like ROUGE are cheap but often misleading, so I'd prefer checking whether the answer contains the key facts from the reference, either with rules or an LLM judge that compares the two against a rubric. I'd check the scorer itself on some examples that humans have labelled, to make sure it agrees with people before trusting it at scale.

More to think about
  • What are the weaknesses of exact match and F1?
  • How would you evaluate a question-answering system?

Go further

Was this lesson clear?