Area 02 · RAG · Concept 19 of 20

Recall@k

Out of all the relevant chunks, how many showed up in the top k results.

11–20 · Production levelPhase 2 · Weeks 7–10≈ 45 min

The idea

You cannot improve retrieval without measuring it. Recall@k asks: for this question, of the chunks that contain the answer, what fraction appear in the top k results? Averaged over a test set, it tells you how often the right information even reaches the model.

To compute it you need labelled data: a set of questions, each with the IDs of the relevant chunks. Thirty to fifty questions written by someone who knows the documents is enough to start.

Pick k to match your pipeline: if you send 5 chunks to the model, Recall@5 is what matters. Run it after every change to chunking, embeddings, hybrid search or reranking.

Why it matters

If the right chunk is not retrieved, no prompt can fix the answer. Recall@k isolates retrieval from generation.

Words to know

  • Relevant setThe chunk IDs labelled as containing the answer.
  • Hit rateShare of questions with at least one relevant chunk in top k.

Try it without code

Label five questions

  1. Write 5 questions about your documents.
  2. For each, find and note the chunk IDs that answer it.
  3. That file is the start of your retrieval eval set.

Use it in code

Recall@k and hit rate over a labelled test set.

python
def recall_at_k(retrieved, relevant, k):
    top = set(retrieved[:k])
    return len(top & set(relevant)) / len(relevant)

def evaluate(test_set, search, k=5):
    recalls, hits = [], 0
    for item in test_set:                       # {"q": "...", "relevant": [ids]}
        got = search(item["q"])                 # returns ranked chunk ids
        r = recall_at_k(got, item["relevant"], k)
        recalls.append(r); hits += r > 0
    return sum(recalls) / len(recalls), hits / len(test_set)

# print(evaluate(TEST_SET, hybrid_search, k=5))
PRACTICE · TICK IT AFTER THIS

Write 30 labelled questions for your PDF chat and report Recall@5 for vector, hybrid, and hybrid + rerank.

Common mistakes

  • Writing test questions by copying sentences from the chunks.
  • Measuring only end answers and never retrieval.

Go deeper