The idea
Recall asks whether the right chunks were retrieved. Context precision asks how many of the retrieved chunks were useful. Sending 8 chunks where only 1 matters wastes tokens and can distract the model.
Measure it by asking a judge, for each retrieved chunk, whether it is relevant to answering the question (optionally given the reference answer). Precision = relevant chunks / retrieved chunks; ranked variants reward relevant chunks appearing early.
Low precision suggests sending fewer chunks, adding a reranker or tightening filters.
It directly affects cost and often quality. Less noise in the prompt usually means better answers.
Words to know
- NoiseRetrieved text that does not help answer the question.
Try it without code
Mark useful chunks
- For 5 questions, list the chunks your system retrieved.
- Mark each useful or not.
- Compute the share that was useful.
Use it in code
Per-chunk usefulness judging.
def context_precision(question, chunks, ask_json):
verdicts = ask_json(
f"Question: {question}\n"
+ "\n".join(f"[{i}] {c}" for i, c in enumerate(chunks))
+ "\nFor each chunk, is it useful for answering? JSON list of true/false in order.")
return sum(verdicts) / len(chunks), verdictsMeasure context precision with top-8 and top-4 retrieval and compare answer quality and cost.
Common mistakes
- Optimising precision so hard that recall collapses.