Area 04 · Evals · Concept 07 of 20

Answer Relevance

Does the answer actually address the question that was asked?

01–10 · The basePhase 4 · Weeks 15–17≈ 45 min

The idea

An answer can be perfectly grounded and still miss the point: the user asked about part-time staff and the answer explains the full-time policy. Answer relevance checks that the response addresses the question’s actual intent and is not padded with off-topic material.

A simple judge prompt works well: given the question and answer, does the answer address the question directly and completely? Some libraries instead generate questions from the answer and compare them to the original question with embeddings.

Track it alongside faithfulness: faithful but irrelevant points to retrieval or prompt issues; relevant but unfaithful points to hallucination.

Why it matters

Users judge your system on whether it answered their question. This metric matches that experience.

Words to know

  • IntentWhat the user actually wants to know or do.

Try it without code

Spot the dodge

  1. Collect 10 answers from a support bot.
  2. Mark any that answer a nearby question instead of the real one.

Use it in code

Relevance judge returning a 0–2 score with reason.

python
RELEVANCE_PROMPT = """Question: {q}
Answer: {a}
Score how well the answer addresses the question:
0 = does not address it, 1 = partly, 2 = directly and completely.
Return JSON: {{"reason": "...", "score": 0|1|2}}"""

# score = ask_json(RELEVANCE_PROMPT.format(q=question, a=answer))["score"]
PRACTICE · TICK IT AFTER THIS

Add relevance to your eval script and list cases with high faithfulness but low relevance.

Common mistakes

  • Penalising correct "I don’t know" answers for unanswerable questions.

Go deeper