If you don't measure retrieval on its own, you can't tell whether a bad answer is the search's fault or the model's.
4 minute read
In short
Evaluate retrieval separately from generation.
Hit rate at k asks if the right passage is in the top k; MRR rewards ranking it higher.
For answers, measure faithfulness, relevance and correctness.
In plain words
A RAG system can fail in two places: retrieval didn't find the right passage, or the model misused it. Measure them separately. For retrieval you need a test set of questions, each paired with the passage that answers it.
Two common retrieval measures. Hit rate at k (also called recall at k): how often is the right passage somewhere in the top k? MRR (mean reciprocal rank): on average, how high is the first right passage? A right passage at rank 1 scores 1, at rank 2 scores 1/2, at rank 3 scores 1/3, and missing scores 0.
Worked example: four test questions
question
rank of the right passage
reciprocal rank
Q1
1
1.000
Q2
3
0.333
Q3
not in top 5
0
Q4
2
0.500
1
Hit rate at 1
Only Q1 has it first.
1 ÷ 4 = 0.25
2
Hit rate at 3
Q1, Q2 and Q4 have it in the top 3.
3 ÷ 4 = 0.75
3
MRR
Average the reciprocal ranks.
(1 + 0.333 + 0 + 0.5) ÷ 4 = 0.458
4
Read the results
Hit rate at 3 is decent, but hit rate at 1 is poor: the right passage is usually found but not on top. A reranker would help. Q3 failed completely; look at why.
Then measure the answers
measure
question it answers
Faithfulness
Is every claim in the answer supported by the retrieved passages?
Answer relevance
Does the answer actually address the question?
Correctness
Does it match the known right answer?
These are usually scored by an AI judge, checked against a sample of human ratings. The Evaluation module covers judges in detail.
Your turn
Hit rate at 10 is 0.95, but hit rate at 1 is 0.40, and you only send the top 2 passages to the model. What should you improve?
What to remember
Build the test set from real user questions. Start with 50 to 100. Measure retrieval and generation separately, and re-run after every change to chunking, embeddings, k or the reranker.
Common mistakes
Measuring only final answers, so you can't tell whether retrieval or generation failed.
Building the test set from questions the system already answers well.
Using an AI judge for faithfulness without checking it against human labels.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
How would you evaluate a RAG system?
I'd evaluate the two halves separately. For retrieval, I'd build a set of real questions with the passages that answer them, and measure hit rate, whether a correct passage appears in the top k, and MRR, how high it ranks. For generation, I'd measure correctness against reference answers, faithfulness, meaning every claim is supported by the retrieved text, and whether citations point to the right sources. An LLM judge can score faithfulness at scale, calibrated against some human labels. I'd also include questions the documents can't answer, to check the system says so instead of guessing. Then in production I'd track user feedback and sample real conversations for review.
More to think about
Explain recall at k and MRR.
What is faithfulness, and how would you measure it?