Area 02 · RAG · Concept 20 of 20

MRR and nDCG

Ranking metrics that reward putting the right chunk near the top of the list.

11–20 · Production levelPhase 2 · Weeks 7–10≈ 45 min

The idea

Recall@k does not care whether the right chunk is first or fifth. Order matters when the model pays more attention to early context or when you show results to users. MRR (Mean Reciprocal Rank) scores each question by 1 / rank of the first relevant result and averages: first place scores 1, second 0.5, tenth 0.1.

nDCG (normalised Discounted Cumulative Gain) handles graded relevance (highly relevant vs partly relevant) and multiple relevant results. It sums relevance discounted by log of position, divided by the best possible ordering, giving a score between 0 and 1.

Use Recall@k to check coverage and MRR or nDCG to check ordering, especially when tuning rerankers.

Why it matters

Rerankers mostly improve order, not coverage. These are the metrics that show it.

Words to know

  • Reciprocal rank1 divided by the position of the first correct result.
  • Graded relevanceRelevance labels with levels, like 0, 1, 2.

Try it without code

Score two rankings

  1. Ranking A has the answer at position 1, ranking B at position 4.
  2. MRR: A = 1.0, B = 0.25.
  3. Repeat for 3 questions and average.

Use it in code

MRR and nDCG@k without libraries.

python
import math

def mrr(results):  # list of (retrieved_ids, relevant_ids)
    total = 0
    for got, rel in results:
        total += next((1 / (i + 1) for i, d in enumerate(got) if d in rel), 0)
    return total / len(results)

def ndcg_at_k(got, grades, k=5):  # grades: {doc_id: 0..3}
    dcg = sum(grades.get(d, 0) / math.log2(i + 2) for i, d in enumerate(got[:k]))
    ideal = sorted(grades.values(), reverse=True)[:k]
    idcg = sum(g / math.log2(i + 2) for i, g in enumerate(ideal))
    return dcg / idcg if idcg else 0

print(mrr([([3, 1, 2], {1}), ([5, 9, 4, 7], {7})]))
print(ndcg_at_k([4, 2, 9], {2: 3, 9: 1}))
PRACTICE · TICK IT AFTER THIS

Add MRR to your eval script and report it next to Recall@5 for each pipeline variant.

Common mistakes

  • Comparing metrics computed on different test sets.
  • Optimising ranking metrics while Recall@k is still low.

Go deeper