What you will have at the end
- golden.jsonl with 50 tagged cases
- Judges validated against your own labels
- Every request traced with cost and latency
- A failing CI check when scores drop
Step by step
- Build the golden set30 real or hand-written questions plus 20 reviewed synthetic ones, tagged by topic and difficulty, with 5 unanswerable.
- Error analysis firstRun the set, read every output, categorise failures and fix the top category before automating anything.
- Write two judgesFaithfulness (claims supported by retrieved context) and answer relevance (0–2). Structured output, temperature 0.
- Validate judgesHand-label 30 outputs, measure agreement, refine rubrics until above 85%.
- Add tracingInstrument retrieval, rerank and generation with Langfuse; record tokens, cost and latency; attach judge scores to traces.
- Eval runner and baselineevals/run.py writes scores.json (Recall@5, faithfulness, relevance, p95 latency, cost per request). Commit a baseline.json from main.
- CI gateGitHub Actions runs a CI subset on pull requests and fails if any metric drops more than your tolerance. Post a summary comment.
- Prove it worksOpen a PR with a deliberately worse prompt, show the check failing, and link it in the README.
DONE WHEN
- One command runs the full eval and prints a table
- Judge agreement with your labels is documented
- A bad PR is blocked by CI
Stretch goals
- Pairwise comparison report for two prompt versions
- Thumbs feedback flowing into a review queue
- Nightly full-suite run with trend chart