Area 04 / 08 · EVL
Evals
Phase 4 · Weeks 15–17How you prove the AI works, and catch it the day it stops working.
Without evals, every prompt change is a guess. This area gives you the tools to measure quality: golden datasets, error analysis, LLM judges, and RAG and agent metrics. The production half puts those measurements into CI, A/B tests, tracing and dashboards, and closes the loop with human labels and user feedback. From here on, every change is measured.
0 of 20 used in code
01–10 · The base
Learn these first.
- 01Golden DatasetsA fixed set of inputs with expected outputs. Every change is tested against it.
- 02Error AnalysisRead real failures, group them by type, fix the biggest group first.
- 03LLM as a JudgeA strong model with a clear rubric grades outputs, so you can score thousands of them.
- 04Pairwise ComparisonThe judge sees two answers and picks the better one. More stable than a 1–10 score.
- 05Rubric ScoringGrade against written criteria (correct, complete, tone), each on a clear scale.
- 06FaithfulnessIs every claim in the answer backed by the retrieved context?
- 07Answer RelevanceDoes the answer actually address the question that was asked?
- 08Context PrecisionHow much of the retrieved context was actually useful for the answer?
- 09Tool Call AccuracyDid the agent pick the right tool, with the right arguments?
- 10Trajectory EvalsJudge the whole path of steps the agent took, not only its final answer.
11–20 · Production level
Come back after you have shipped something.
- 11Regression Evals in CIRun the eval suite on every prompt or code change and block the change if scores drop.
- 12A/B Testing PromptsSend two prompt versions to real users and compare the numbers.
- 13Human LabelingPeople grade a sample of outputs. This is the truth you check your judges against.
- 14Synthetic Test DataHave an LLM generate varied test cases, edge cases included, then review them yourself.
- 15TracingRecord every step of a request (prompts, tool calls, time, tokens) so you can debug it later.
- 16LangfuseOpen-source LLM observability: traces, prompt management, evals and cost tracking.
- 17OpenTelemetry for LLMsThe open tracing standard, with conventions for LLM calls, so traces work across tools.
- 18P95 Latency95% of requests are faster than this. It shows the slow tail that an average hides.
- 19Cost per RequestTotal token + infra spend divided by requests. Track it per feature.
- 20User Feedback LoopsThumbs up/down and user corrections flow back into your test datasets.
BUILD THIS
Add 50 golden questions to your RAG app, an LLM judge for faithfulness and relevance, Langfuse tracing, and a CI check that fails when scores drop.
Project guide →