The idea
Recall@k does not care whether the right chunk is first or fifth. Order matters when the model pays more attention to early context or when you show results to users. MRR (Mean Reciprocal Rank) scores each question by 1 / rank of the first relevant result and averages: first place scores 1, second 0.5, tenth 0.1.
nDCG (normalised Discounted Cumulative Gain) handles graded relevance (highly relevant vs partly relevant) and multiple relevant results. It sums relevance discounted by log of position, divided by the best possible ordering, giving a score between 0 and 1.
Use Recall@k to check coverage and MRR or nDCG to check ordering, especially when tuning rerankers.
Rerankers mostly improve order, not coverage. These are the metrics that show it.
Words to know
- Reciprocal rank1 divided by the position of the first correct result.
- Graded relevanceRelevance labels with levels, like 0, 1, 2.
Try it without code
Score two rankings
- Ranking A has the answer at position 1, ranking B at position 4.
- MRR: A = 1.0, B = 0.25.
- Repeat for 3 questions and average.
Use it in code
MRR and nDCG@k without libraries.
import math
def mrr(results): # list of (retrieved_ids, relevant_ids)
total = 0
for got, rel in results:
total += next((1 / (i + 1) for i, d in enumerate(got) if d in rel), 0)
return total / len(results)
def ndcg_at_k(got, grades, k=5): # grades: {doc_id: 0..3}
dcg = sum(grades.get(d, 0) / math.log2(i + 2) for i, d in enumerate(got[:k]))
ideal = sorted(grades.values(), reverse=True)[:k]
idcg = sum(g / math.log2(i + 2) for i, g in enumerate(ideal))
return dcg / idcg if idcg else 0
print(mrr([([3, 1, 2], {1}), ([5, 9, 4, 7], {7})]))
print(ndcg_at_k([4, 2, 9], {2: 3, 9: 1}))Add MRR to your eval script and report it next to Recall@5 for each pipeline variant.
Common mistakes
- Comparing metrics computed on different test sets.
- Optimising ranking metrics while Recall@k is still low.