The idea
Offline evals tell you a change is safe; only real users tell you it is better for them. An A/B test sends a random share of traffic to version B and compares business and quality metrics: task completion, thumbs up rate, follow-up questions, escalations to humans, cost.
Assign users (not requests) to variants consistently, log the variant with every request, decide the metric and sample size before starting, and run long enough to see real differences.
Prompt versions should be identified like code versions so results map exactly to the text that ran.
It connects AI quality to outcomes the business cares about.
Words to know
- VariantOne version under test.
- Statistical significanceConfidence the difference is not random noise.
Try it without code
Design a test
- Pick a prompt change.
- Write the primary metric, the expected change, and how many users you need (use an online sample size calculator).
Use it in code
Stable variant assignment and a two-proportion z-test.
import hashlib, math
def variant(user_id, split=0.5):
h = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 10_000
return "B" if h < split * 10_000 else "A"
def z_test(success_a, n_a, success_b, n_b):
p_a, p_b = success_a / n_a, success_b / n_b
p = (success_a + success_b) / (n_a + n_b)
z = (p_b - p_a) / math.sqrt(p * (1 - p) * (1 / n_a + 1 / n_b))
return p_a, p_b, z # |z| > 1.96 is roughly significant at 95%
print(variant("user-123"), z_test(410, 1000, 455, 1000))Add variant assignment to your chatbot, log variant + thumbs feedback, and simulate results to practise the analysis.
Common mistakes
- Assigning per request so users bounce between versions.
- Stopping the test the moment B looks better.