The idea
Before launch you may have little real data. An LLM can generate test inputs quickly: questions about your documents, user messages in different tones, typos, other languages, adversarial requests. This fills gaps in coverage.
Generate along explicit dimensions (persona × intent × difficulty) instead of "give me 50 questions", which produces samey output. Then review and edit: drop unrealistic cases, fix wrong expected answers, and tag them as synthetic.
Replace or supplement synthetic cases with real ones as soon as you have production traffic.
It lets you build a reasonable eval set in hours instead of weeks, especially for edge cases.
Words to know
- DimensionsThe axes you vary to get diverse cases.
Try it without code
Generate on a grid
- Pick 3 personas, 3 intents and 2 difficulty levels.
- Ask a chat app for one realistic message per combination (18 total).
- Delete the unrealistic ones.
Use it in code
Generate cases across a grid of dimensions.
import itertools, json, anthropic
client = anthropic.Anthropic()
personas = ["new customer", "angry long-time customer", "non-native English speaker"]
intents = ["refund", "password reset", "cancel plan"]
levels = ["simple", "ambiguous"]
cases = []
for p, i, l in itertools.product(personas, intents, levels):
m = client.messages.create(model="claude-haiku-5-5", max_tokens=150, temperature=1,
messages=[{"role": "user", "content": f"Write one realistic {l} support message from a {p} about {i}. Message only."}])
cases.append({"input": m.content[0].text.strip(), "tags": [p, i, l, "synthetic"]})
json.dump(cases, open("synthetic.json", "w"), indent=2)
print(len(cases), "cases - now review them by hand")Generate 40 synthetic questions for your RAG app, keep the best 20 after review, and add them to the golden set.
Common mistakes
- Using synthetic cases unreviewed.
- Generating expected answers with the same system you are testing.