Area 04 · Evals · Concept 04 of 20

Pairwise Comparison

The judge sees two answers and picks the better one. More stable than a 1–10 score.

01–10 · The basePhase 4 · Weeks 15–17≈ 45 min

The idea

Absolute scores drift: the same answer can get a 6 or an 8 depending on the run. Comparing two answers side by side is easier for humans and models alike. Pairwise evaluation shows the judge answers A and B for the same input and asks which is better, or whether they tie.

Run each pair twice with A and B swapped to cancel position bias (judges often prefer the first or the longer answer). Count wins, losses and ties to compare two prompt versions or models.

It answers "is version 2 better than version 1?" well. It does not tell you whether either is good enough; pair it with absolute pass/fail checks.

Why it matters

It is the most reliable way to decide between two prompts or models on subjective quality.

Words to know

  • Position biasA judge preferring whichever answer is shown first.
  • Win rateShare of comparisons a version wins.

Try it without code

Blind A/B

  1. Generate two answers to the same question with different prompts.
  2. Ask a friend which is better without telling them which is which.
  3. Swap order and ask again.

Use it in code

Pairwise judge with order swapping.

python
import anthropic
client = anthropic.Anthropic()

def prefer(question, a, b):
    msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=5, temperature=0,
        messages=[{"role": "user", "content":
            f"Question: {question}\n<answer_1>{a}</answer_1>\n<answer_2>{b}</answer_2>\n"
            "Which answer is more helpful and correct? Reply 1, 2 or TIE."}])
    return msg.content[0].text.strip()

def compare(question, v1, v2):
    first, second = prefer(question, v1, v2), prefer(question, v2, v1)
    if first == "1" and second == "2": return "v1"
    if first == "2" and second == "1": return "v2"
    return "tie"
PRACTICE · TICK IT AFTER THIS

Compare two system prompts for your chatbot on 30 questions and report win/tie/loss.

Common mistakes

  • Not swapping order.
  • Letting length decide; tell the judge not to reward length.

Go deeper