Area 04 · Evals · Concept 12 of 20

A/B Testing Prompts

Send two prompt versions to real users and compare the numbers.

11–20 · Production levelPhase 4 · Weeks 15–17≈ 1 hour

The idea

Offline evals tell you a change is safe; only real users tell you it is better for them. An A/B test sends a random share of traffic to version B and compares business and quality metrics: task completion, thumbs up rate, follow-up questions, escalations to humans, cost.

Assign users (not requests) to variants consistently, log the variant with every request, decide the metric and sample size before starting, and run long enough to see real differences.

Prompt versions should be identified like code versions so results map exactly to the text that ran.

Why it matters

It connects AI quality to outcomes the business cares about.

Words to know

  • VariantOne version under test.
  • Statistical significanceConfidence the difference is not random noise.

Try it without code

Design a test

  1. Pick a prompt change.
  2. Write the primary metric, the expected change, and how many users you need (use an online sample size calculator).

Use it in code

Stable variant assignment and a two-proportion z-test.

python
import hashlib, math

def variant(user_id, split=0.5):
    h = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 10_000
    return "B" if h < split * 10_000 else "A"

def z_test(success_a, n_a, success_b, n_b):
    p_a, p_b = success_a / n_a, success_b / n_b
    p = (success_a + success_b) / (n_a + n_b)
    z = (p_b - p_a) / math.sqrt(p * (1 - p) * (1 / n_a + 1 / n_b))
    return p_a, p_b, z          # |z| > 1.96 is roughly significant at 95%

print(variant("user-123"), z_test(410, 1000, 455, 1000))
PRACTICE · TICK IT AFTER THIS

Add variant assignment to your chatbot, log variant + thumbs feedback, and simulate results to practise the analysis.

Common mistakes

  • Assigning per request so users bounce between versions.
  • Stopping the test the moment B looks better.

Go deeper