Area 04 · Evals · Build this

Eval suite and CI gate for your RAG app

Turn your Phase 2 RAG app into something you can change with confidence: a 50-question golden set, judges for faithfulness and relevance, tracing in Langfuse, and a GitHub check that blocks regressions.

Phase 4 · Weeks 15–17

What you will have at the end

  • golden.jsonl with 50 tagged cases
  • Judges validated against your own labels
  • Every request traced with cost and latency
  • A failing CI check when scores drop

Step by step

  1. Build the golden set30 real or hand-written questions plus 20 reviewed synthetic ones, tagged by topic and difficulty, with 5 unanswerable.
  2. Error analysis firstRun the set, read every output, categorise failures and fix the top category before automating anything.
  3. Write two judgesFaithfulness (claims supported by retrieved context) and answer relevance (0–2). Structured output, temperature 0.
  4. Validate judgesHand-label 30 outputs, measure agreement, refine rubrics until above 85%.
  5. Add tracingInstrument retrieval, rerank and generation with Langfuse; record tokens, cost and latency; attach judge scores to traces.
  6. Eval runner and baselineevals/run.py writes scores.json (Recall@5, faithfulness, relevance, p95 latency, cost per request). Commit a baseline.json from main.
  7. CI gateGitHub Actions runs a CI subset on pull requests and fails if any metric drops more than your tolerance. Post a summary comment.
  8. Prove it worksOpen a PR with a deliberately worse prompt, show the check failing, and link it in the README.
DONE WHEN
  • One command runs the full eval and prints a table
  • Judge agreement with your labels is documented
  • A bad PR is blocked by CI

Stretch goals

  • Pairwise comparison report for two prompt versions
  • Thumbs feedback flowing into a review queue
  • Nightly full-suite run with trend chart