The idea
Prompt and model changes break things silently. A regression eval runs your golden set and judges automatically on every pull request, compares scores with the main branch baseline, and fails the check if a metric drops beyond a tolerance.
Keep the CI suite fast and cheap: a representative subset (50–100 cases), cheap judges where possible, and caching of unchanged results. Run the full suite nightly.
Post a summary table to the pull request so reviewers see exactly which cases changed.
It makes AI changes as safe to merge as normal code changes.
Words to know
- BaselineThe scores of the current main branch.
- ToleranceHow much a score may drop before the check fails.
Try it without code
Pick thresholds
- Write your current scores for 3 metrics.
- Decide the drop that should block a merge for each.
Use it in code
GitHub Actions job that runs evals and fails on regression.
# .github/workflows/evals.yml
name: evals
on: [pull_request]
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install -r requirements.txt
- run: python evals/run.py --subset ci --out scores.json
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- run: python evals/compare.py scores.json evals/baseline.json --max-drop 0.03Add an eval workflow to your RAG repo that fails when faithfulness or Recall@5 drops more than 3 points.
Common mistakes
- A CI suite so slow people disable it.
- Updating the baseline automatically on every merge without review.