The idea
Absolute scores drift: the same answer can get a 6 or an 8 depending on the run. Comparing two answers side by side is easier for humans and models alike. Pairwise evaluation shows the judge answers A and B for the same input and asks which is better, or whether they tie.
Run each pair twice with A and B swapped to cancel position bias (judges often prefer the first or the longer answer). Count wins, losses and ties to compare two prompt versions or models.
It answers "is version 2 better than version 1?" well. It does not tell you whether either is good enough; pair it with absolute pass/fail checks.
It is the most reliable way to decide between two prompts or models on subjective quality.
Words to know
- Position biasA judge preferring whichever answer is shown first.
- Win rateShare of comparisons a version wins.
Try it without code
Blind A/B
- Generate two answers to the same question with different prompts.
- Ask a friend which is better without telling them which is which.
- Swap order and ask again.
Use it in code
Pairwise judge with order swapping.
import anthropic
client = anthropic.Anthropic()
def prefer(question, a, b):
msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=5, temperature=0,
messages=[{"role": "user", "content":
f"Question: {question}\n<answer_1>{a}</answer_1>\n<answer_2>{b}</answer_2>\n"
"Which answer is more helpful and correct? Reply 1, 2 or TIE."}])
return msg.content[0].text.strip()
def compare(question, v1, v2):
first, second = prefer(question, v1, v2), prefer(question, v2, v1)
if first == "1" and second == "2": return "v1"
if first == "2" and second == "1": return "v2"
return "tie"Compare two system prompts for your chatbot on 30 questions and report win/tie/loss.
Common mistakes
- Not swapping order.
- Letting length decide; tell the judge not to reward length.