The idea
You cannot improve retrieval without measuring it. Recall@k asks: for this question, of the chunks that contain the answer, what fraction appear in the top k results? Averaged over a test set, it tells you how often the right information even reaches the model.
To compute it you need labelled data: a set of questions, each with the IDs of the relevant chunks. Thirty to fifty questions written by someone who knows the documents is enough to start.
Pick k to match your pipeline: if you send 5 chunks to the model, Recall@5 is what matters. Run it after every change to chunking, embeddings, hybrid search or reranking.
If the right chunk is not retrieved, no prompt can fix the answer. Recall@k isolates retrieval from generation.
Words to know
- Relevant setThe chunk IDs labelled as containing the answer.
- Hit rateShare of questions with at least one relevant chunk in top k.
Try it without code
Label five questions
- Write 5 questions about your documents.
- For each, find and note the chunk IDs that answer it.
- That file is the start of your retrieval eval set.
Use it in code
Recall@k and hit rate over a labelled test set.
def recall_at_k(retrieved, relevant, k):
top = set(retrieved[:k])
return len(top & set(relevant)) / len(relevant)
def evaluate(test_set, search, k=5):
recalls, hits = [], 0
for item in test_set: # {"q": "...", "relevant": [ids]}
got = search(item["q"]) # returns ranked chunk ids
r = recall_at_k(got, item["relevant"], k)
recalls.append(r); hits += r > 0
return sum(recalls) / len(recalls), hits / len(test_set)
# print(evaluate(TEST_SET, hybrid_search, k=5))Write 30 labelled questions for your PDF chat and report Recall@5 for vector, hybrid, and hybrid + rerank.
Common mistakes
- Writing test questions by copying sentences from the chunks.
- Measuring only end answers and never retrieval.