Learn · Module 11 · Evaluation · Lesson 2 of 5
Scoring against references
When you know the right answer, compare against it. The question is how strict to be.
- Exact match is right for structured answers like IDs, labels and numbers.
- Token F1 gives partial credit for overlapping words in short answers.
- For long free text, word overlap is a poor guide; use a judge with the reference.
In plain words
A reference is a known correct answer. Exact match gives 1 only if the output equals it after simple cleanup (lowercase, trim spaces). Token F1 gives partial credit for overlapping words, combining precision (how much of the output is in the reference) and recall (how much of the reference is in the output).
Worked example: who led the Salt March?
The reference is 'Mahatma Gandhi'.
| model's answer | exact match | precision | recall | F1 |
|---|---|---|---|---|
| Mahatma Gandhi | 1 | 1.00 | 1.00 | 1.000 |
| Gandhi | 0 | 1.00 | 0.50 | 0.667 |
| Mahatma Gandhi led the movement | 0 | 0.40 | 1.00 | 0.571 |
| it was Jawaharlal Nehru | 0 | 0 | 0 | 0 |
- 1'Gandhi' is right but scores 0 on exact match
Exact match is too strict for free text.
- 2F1 gives fair partial credit
'Gandhi' matches 1 of 1 output words (precision 1.0) and 1 of 2 reference words (recall 0.5).
F1 = 2 × 1.0 × 0.5 ÷ 1.5 = 0.667
- 3But F1 punishes extra words
A correct, longer sentence scores lower than the bare name.
You're grading a model that extracts invoice numbers like 'INV-20931'. Which scoring is right?
What to remember
Use exact match for structured answers: IDs, labels, numbers, JSON fields. Use F1 or similar overlap scores for short free-text answers. For longer, open-ended answers, overlap scores stop meaning much, and you'll need an AI judge with the reference, the next lesson.
Common mistakes
- Using exact match for free-text answers, which marks correct paraphrases as wrong.
- Trusting overlap scores like BLEU or ROUGE for open-ended answers; they reward similar words, not correctness.
- Writing reference answers so narrow that other valid answers fail.
Answer in your own words, out loud or on paper, before you open the model answer.
How would you score answers when you have a reference answer?
It depends on the kind of answer. For short factual or structured answers, like a category, a number or an ID, I'd use exact match after normalising case, spacing and formats. For extracted fields, I'd score each field separately so I can see which ones fail. For free-text answers, word-overlap scores like ROUGE are cheap but often misleading, so I'd prefer checking whether the answer contains the key facts from the reference, either with rules or an LLM judge that compares the two against a rubric. I'd check the scorer itself on some examples that humans have labelled, to make sure it agrees with people before trusting it at scale.
- What are the weaknesses of exact match and F1?
- How would you evaluate a question-answering system?
Go further
- ArticleTask-Specific LLM Evals that Do & Don't WorkEugene Yan
- PaperBERTScore: Evaluating Text Generation with BERTZhang et al., 2019