Area 04 · Evals · Concept 14 of 20

Synthetic Test Data

Have an LLM generate varied test cases, edge cases included, then review them yourself.

11–20 · Production levelPhase 4 · Weeks 15–17≈ 1 hour

The idea

Before launch you may have little real data. An LLM can generate test inputs quickly: questions about your documents, user messages in different tones, typos, other languages, adversarial requests. This fills gaps in coverage.

Generate along explicit dimensions (persona × intent × difficulty) instead of "give me 50 questions", which produces samey output. Then review and edit: drop unrealistic cases, fix wrong expected answers, and tag them as synthetic.

Replace or supplement synthetic cases with real ones as soon as you have production traffic.

Why it matters

It lets you build a reasonable eval set in hours instead of weeks, especially for edge cases.

Words to know

  • DimensionsThe axes you vary to get diverse cases.

Try it without code

Generate on a grid

  1. Pick 3 personas, 3 intents and 2 difficulty levels.
  2. Ask a chat app for one realistic message per combination (18 total).
  3. Delete the unrealistic ones.

Use it in code

Generate cases across a grid of dimensions.

python
import itertools, json, anthropic
client = anthropic.Anthropic()

personas = ["new customer", "angry long-time customer", "non-native English speaker"]
intents = ["refund", "password reset", "cancel plan"]
levels = ["simple", "ambiguous"]

cases = []
for p, i, l in itertools.product(personas, intents, levels):
    m = client.messages.create(model="claude-haiku-5-5", max_tokens=150, temperature=1,
        messages=[{"role": "user", "content": f"Write one realistic {l} support message from a {p} about {i}. Message only."}])
    cases.append({"input": m.content[0].text.strip(), "tags": [p, i, l, "synthetic"]})
json.dump(cases, open("synthetic.json", "w"), indent=2)
print(len(cases), "cases - now review them by hand")
PRACTICE · TICK IT AFTER THIS

Generate 40 synthetic questions for your RAG app, keep the best 20 after review, and add them to the golden set.

Common mistakes

  • Using synthetic cases unreviewed.
  • Generating expected answers with the same system you are testing.

Go deeper