The idea
An answer can be perfectly grounded and still miss the point: the user asked about part-time staff and the answer explains the full-time policy. Answer relevance checks that the response addresses the question’s actual intent and is not padded with off-topic material.
A simple judge prompt works well: given the question and answer, does the answer address the question directly and completely? Some libraries instead generate questions from the answer and compare them to the original question with embeddings.
Track it alongside faithfulness: faithful but irrelevant points to retrieval or prompt issues; relevant but unfaithful points to hallucination.
Users judge your system on whether it answered their question. This metric matches that experience.
Words to know
- IntentWhat the user actually wants to know or do.
Try it without code
Spot the dodge
- Collect 10 answers from a support bot.
- Mark any that answer a nearby question instead of the real one.
Use it in code
Relevance judge returning a 0–2 score with reason.
RELEVANCE_PROMPT = """Question: {q}
Answer: {a}
Score how well the answer addresses the question:
0 = does not address it, 1 = partly, 2 = directly and completely.
Return JSON: {{"reason": "...", "score": 0|1|2}}"""
# score = ask_json(RELEVANCE_PROMPT.format(q=question, a=answer))["score"]Add relevance to your eval script and list cases with high faithfulness but low relevance.
Common mistakes
- Penalising correct "I don’t know" answers for unanswerable questions.