Area 04 · Evals · Concept 06 of 20

Faithfulness

Is every claim in the answer backed by the retrieved context?

01–10 · The basePhase 4 · Weeks 15–17≈ 1 hour

The idea

In RAG, an answer can be fluent and relevant yet include facts not in the retrieved sources. Faithfulness (or groundedness) measures whether every claim is supported by the context the model was given.

The usual method: split the answer into individual claims, then for each claim ask a judge whether the context supports it. Faithfulness = supported claims / total claims.

Low faithfulness means hallucination or the model using its own memory. Fix it with stricter grounding instructions, better citations, or better retrieval if the right context was missing.

Why it matters

It is the core trust metric for any RAG product.

Words to know

  • ClaimOne checkable statement in an answer.

Try it without code

Check claims by hand

  1. Take one RAG answer and its sources.
  2. List each claim.
  3. Mark each as supported or not, and compute the ratio.

Use it in code

Claim extraction plus per-claim support check.

python
import json, anthropic
client = anthropic.Anthropic()

def ask_json(prompt):
    m = client.messages.create(model="claude-sonnet-5-5", max_tokens=800, temperature=0,
                               messages=[{"role": "user", "content": prompt}])
    return json.loads(m.content[0].text)

def faithfulness(answer, context):
    claims = ask_json(f"List every factual claim in this answer as a JSON list of strings.\n{answer}")
    verdicts = ask_json(
        f"<context>{context}</context>\nFor each claim, is it supported by the context? "
        f"Return a JSON list of true/false in order.\nClaims: {json.dumps(claims)}")
    return sum(verdicts) / max(len(claims), 1), list(zip(claims, verdicts))
PRACTICE · TICK IT AFTER THIS

Compute faithfulness for your 50 golden questions and read the 5 lowest-scoring answers.

Common mistakes

  • Judging against the full document instead of what was actually retrieved.
  • Counting refusals as unfaithful.

Go deeper