The idea
A golden dataset is your test suite for AI behaviour: real-looking inputs, each with the expected output or the criteria a good output must meet. It stays fixed so results are comparable over time.
Build it from real usage where possible (logs, support tickets, user questions), cover common cases and known hard cases, and include inputs that should be refused or answered with "I don’t know". Start with 30–50 items; grow it every time you find a new failure.
Store it in version control as JSONL or CSV with an ID, input, expected answer or rubric, and tags (topic, difficulty) so you can slice results.
It is the foundation of every other eval. Without a fixed set, "it seems better" is the best you can say.
Words to know
- Test caseOne input with its expected outcome.
- SliceA subset of cases sharing a tag, like "billing" or "hard".
Try it without code
Write ten cases
- Open a spreadsheet with columns id, input, expected, tags.
- Write 10 cases for an AI feature you know, including 2 that should be refused.
Use it in code
Load a JSONL golden set and run a system over it.
import json
# golden.jsonl lines: {"id": "q01", "input": "...", "expected": "...", "tags": ["billing"]}
cases = [json.loads(l) for l in open("golden.jsonl")]
def run(system_fn):
rows = []
for c in cases:
out = system_fn(c["input"])
rows.append({**c, "output": out})
json.dump(rows, open("run_latest.json", "w"), indent=2)
return rows
# rows = run(my_rag_answer)Create golden.jsonl with 30 cases for your RAG app, tagged by topic and difficulty.
Common mistakes
- Only easy happy-path cases.
- Changing the set between comparisons.
- Writing cases by copying the documents word for word.