Area 07 · Deployment · Concept 06 of 20

Eval Gates in CI

The deploy stops if eval scores fall below your threshold.

01–10 · The basePhase 5 · Weeks 18–20≈ 1 hour

The idea

Regression evals on pull requests catch most problems, but the deploy pipeline needs its own gate: before promoting a build to production, run the eval suite against the built artefact (or staging) and stop if scores are below absolute thresholds or worse than the current production version.

Gate on the metrics that matter: faithfulness, task success, red-team attack success rate, p95 latency and cost per request. A cheaper prompt that drops quality should not ship automatically.

Make the gate visible: publish the scores as a job summary and keep a history so you can see trends across releases.

Why it matters

It turns "we measure quality" into "we cannot ship worse quality by accident".

Words to know

  • GateA pipeline step that must pass for the release to continue.

Try it without code

Set thresholds

  1. Write minimum acceptable values for 4 metrics.
  2. Decide which are absolute and which are relative to production.

Use it in code

A gate script that fails the job and writes a summary.

python
import json, os, sys

scores = json.load(open("scores.json"))
prod = json.load(open("prod_scores.json"))
RULES = {"faithfulness": ("min", 0.90), "task_success": ("min", 0.85),
         "attack_success": ("max", 0.05), "p95_latency_s": ("max", 4.0)}

failed, lines = [], ["| metric | new | prod | rule |", "|---|---|---|---|"]
for m, (kind, limit) in RULES.items():
    v = scores[m]
    ok = v >= limit if kind == "min" else v <= limit
    ok = ok and (v >= prod[m] - 0.02 if kind == "min" else v <= prod[m] * 1.10)
    lines.append(f"| {m} | {v} | {prod[m]} | {kind} {limit} |{'' if ok else ' FAIL'}")
    if not ok: failed.append(m)

open(os.environ.get("GITHUB_STEP_SUMMARY", "/dev/stdout"), "a").write("\n".join(lines) + "\n")
sys.exit(1 if failed else 0)
PRACTICE · TICK IT AFTER THIS

Add an eval gate job between staging and production deploy and prove it blocks a degraded prompt.

Common mistakes

  • Gates so slow that people bypass them.
  • Thresholds never revisited as the product improves.

Go deeper