The idea
In RAG, an answer can be fluent and relevant yet include facts not in the retrieved sources. Faithfulness (or groundedness) measures whether every claim is supported by the context the model was given.
The usual method: split the answer into individual claims, then for each claim ask a judge whether the context supports it. Faithfulness = supported claims / total claims.
Low faithfulness means hallucination or the model using its own memory. Fix it with stricter grounding instructions, better citations, or better retrieval if the right context was missing.
It is the core trust metric for any RAG product.
Words to know
- ClaimOne checkable statement in an answer.
Try it without code
Check claims by hand
- Take one RAG answer and its sources.
- List each claim.
- Mark each as supported or not, and compute the ratio.
Use it in code
Claim extraction plus per-claim support check.
import json, anthropic
client = anthropic.Anthropic()
def ask_json(prompt):
m = client.messages.create(model="claude-sonnet-5-5", max_tokens=800, temperature=0,
messages=[{"role": "user", "content": prompt}])
return json.loads(m.content[0].text)
def faithfulness(answer, context):
claims = ask_json(f"List every factual claim in this answer as a JSON list of strings.\n{answer}")
verdicts = ask_json(
f"<context>{context}</context>\nFor each claim, is it supported by the context? "
f"Return a JSON list of true/false in order.\nClaims: {json.dumps(claims)}")
return sum(verdicts) / max(len(claims), 1), list(zip(claims, verdicts))Compute faithfulness for your 50 golden questions and read the 5 lowest-scoring answers.
Common mistakes
- Judging against the full document instead of what was actually retrieved.
- Counting refusals as unfaithful.