Area 04 · Evals · Concept 08 of 20

Context Precision

How much of the retrieved context was actually useful for the answer?

01–10 · The basePhase 4 · Weeks 15–17≈ 45 min

The idea

Recall asks whether the right chunks were retrieved. Context precision asks how many of the retrieved chunks were useful. Sending 8 chunks where only 1 matters wastes tokens and can distract the model.

Measure it by asking a judge, for each retrieved chunk, whether it is relevant to answering the question (optionally given the reference answer). Precision = relevant chunks / retrieved chunks; ranked variants reward relevant chunks appearing early.

Low precision suggests sending fewer chunks, adding a reranker or tightening filters.

Why it matters

It directly affects cost and often quality. Less noise in the prompt usually means better answers.

Words to know

  • NoiseRetrieved text that does not help answer the question.

Try it without code

Mark useful chunks

  1. For 5 questions, list the chunks your system retrieved.
  2. Mark each useful or not.
  3. Compute the share that was useful.

Use it in code

Per-chunk usefulness judging.

python
def context_precision(question, chunks, ask_json):
    verdicts = ask_json(
        f"Question: {question}\n"
        + "\n".join(f"[{i}] {c}" for i, c in enumerate(chunks))
        + "\nFor each chunk, is it useful for answering? JSON list of true/false in order.")
    return sum(verdicts) / len(chunks), verdicts
PRACTICE · TICK IT AFTER THIS

Measure context precision with top-8 and top-4 retrieval and compare answer quality and cost.

Common mistakes

  • Optimising precision so hard that recall collapses.

Go deeper