The idea
Regression evals on pull requests catch most problems, but the deploy pipeline needs its own gate: before promoting a build to production, run the eval suite against the built artefact (or staging) and stop if scores are below absolute thresholds or worse than the current production version.
Gate on the metrics that matter: faithfulness, task success, red-team attack success rate, p95 latency and cost per request. A cheaper prompt that drops quality should not ship automatically.
Make the gate visible: publish the scores as a job summary and keep a history so you can see trends across releases.
It turns "we measure quality" into "we cannot ship worse quality by accident".
Words to know
- GateA pipeline step that must pass for the release to continue.
Try it without code
Set thresholds
- Write minimum acceptable values for 4 metrics.
- Decide which are absolute and which are relative to production.
Use it in code
A gate script that fails the job and writes a summary.
import json, os, sys
scores = json.load(open("scores.json"))
prod = json.load(open("prod_scores.json"))
RULES = {"faithfulness": ("min", 0.90), "task_success": ("min", 0.85),
"attack_success": ("max", 0.05), "p95_latency_s": ("max", 4.0)}
failed, lines = [], ["| metric | new | prod | rule |", "|---|---|---|---|"]
for m, (kind, limit) in RULES.items():
v = scores[m]
ok = v >= limit if kind == "min" else v <= limit
ok = ok and (v >= prod[m] - 0.02 if kind == "min" else v <= prod[m] * 1.10)
lines.append(f"| {m} | {v} | {prod[m]} | {kind} {limit} |{'' if ok else ' FAIL'}")
if not ok: failed.append(m)
open(os.environ.get("GITHUB_STEP_SUMMARY", "/dev/stdout"), "a").write("\n".join(lines) + "\n")
sys.exit(1 if failed else 0)Add an eval gate job between staging and production deploy and prove it blocks a degraded prompt.
Common mistakes
- Gates so slow that people bypass them.
- Thresholds never revisited as the product improves.