Area 04 · Evals · Concept 11 of 20

Regression Evals in CI

Run the eval suite on every prompt or code change and block the change if scores drop.

11–20 · Production levelPhase 4 · Weeks 15–17≈ 1.5 hours

The idea

Prompt and model changes break things silently. A regression eval runs your golden set and judges automatically on every pull request, compares scores with the main branch baseline, and fails the check if a metric drops beyond a tolerance.

Keep the CI suite fast and cheap: a representative subset (50–100 cases), cheap judges where possible, and caching of unchanged results. Run the full suite nightly.

Post a summary table to the pull request so reviewers see exactly which cases changed.

Why it matters

It makes AI changes as safe to merge as normal code changes.

Words to know

  • BaselineThe scores of the current main branch.
  • ToleranceHow much a score may drop before the check fails.

Try it without code

Pick thresholds

  1. Write your current scores for 3 metrics.
  2. Decide the drop that should block a merge for each.

Use it in code

GitHub Actions job that runs evals and fails on regression.

yaml
# .github/workflows/evals.yml
name: evals
on: [pull_request]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: python evals/run.py --subset ci --out scores.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - run: python evals/compare.py scores.json evals/baseline.json --max-drop 0.03
PRACTICE · TICK IT AFTER THIS

Add an eval workflow to your RAG repo that fails when faithfulness or Recall@5 drops more than 3 points.

Common mistakes

  • A CI suite so slow people disable it.
  • Updating the baseline automatically on every merge without review.

Go deeper