Area 04 / 08 · EVL

Evals

Phase 4 · Weeks 15–17

How you prove the AI works, and catch it the day it stops working.

Without evals, every prompt change is a guess. This area gives you the tools to measure quality: golden datasets, error analysis, LLM judges, and RAG and agent metrics. The production half puts those measurements into CI, A/B tests, tracing and dashboards, and closes the loop with human labels and user feedback. From here on, every change is measured.

0 of 20 used in code

01–10 · The base

Learn these first.

  • 01Golden DatasetsA fixed set of inputs with expected outputs. Every change is tested against it.
  • 02Error AnalysisRead real failures, group them by type, fix the biggest group first.
  • 03LLM as a JudgeA strong model with a clear rubric grades outputs, so you can score thousands of them.
  • 04Pairwise ComparisonThe judge sees two answers and picks the better one. More stable than a 1–10 score.
  • 05Rubric ScoringGrade against written criteria (correct, complete, tone), each on a clear scale.
  • 06FaithfulnessIs every claim in the answer backed by the retrieved context?
  • 07Answer RelevanceDoes the answer actually address the question that was asked?
  • 08Context PrecisionHow much of the retrieved context was actually useful for the answer?
  • 09Tool Call AccuracyDid the agent pick the right tool, with the right arguments?
  • 10Trajectory EvalsJudge the whole path of steps the agent took, not only its final answer.

11–20 · Production level

Come back after you have shipped something.

  • 11Regression Evals in CIRun the eval suite on every prompt or code change and block the change if scores drop.
  • 12A/B Testing PromptsSend two prompt versions to real users and compare the numbers.
  • 13Human LabelingPeople grade a sample of outputs. This is the truth you check your judges against.
  • 14Synthetic Test DataHave an LLM generate varied test cases, edge cases included, then review them yourself.
  • 15TracingRecord every step of a request (prompts, tool calls, time, tokens) so you can debug it later.
  • 16LangfuseOpen-source LLM observability: traces, prompt management, evals and cost tracking.
  • 17OpenTelemetry for LLMsThe open tracing standard, with conventions for LLM calls, so traces work across tools.
  • 18P95 Latency95% of requests are faster than this. It shows the slow tail that an average hides.
  • 19Cost per RequestTotal token + infra spend divided by requests. Track it per feature.
  • 20User Feedback LoopsThumbs up/down and user corrections flow back into your test datasets.
BUILD THIS

Add 50 golden questions to your RAG app, an LLM judge for faithfulness and relevance, Langfuse tracing, and a CI check that fails when scores drop.

Project guide →