What you will have at the end
- redteam.jsonl with 20+ attacks, goals and expected safe behaviour
- A before/after report of attack success rate and false refusals
- Guardrails, output validation, tool permissions and an approval step in code
Step by step
- Threat modelFill in the OWASP LLM Top 10 table for your agent and list 5 attacker goals.
- Write the attacks20 attacks across goals: direct injection, indirect injection in a fetched page, prompt extraction, jailbreak, data exfiltration via email tool.
- Baseline runRun all attacks, judge success, and record the attack success rate plus 30 normal tasks as a quality baseline.
- Input guardrailsLength and secret checks, an injection classifier, and untrusted-content wrapping for fetched pages.
- Least privilegeNarrow tools, read-only by default, identity from the session, and drop outbound tools after untrusted content is read.
- Output validation and approvalValidated final output, canary check for prompt leakage, and human approval for send_email.
- Audit logLog every tool call, block and approval as structured events.
- Re-run and reportRun attacks and normal tasks again. Report attack success rate, false refusals and added latency in SECURITY.md.
DONE WHEN
- Attack success rate is documented before and after
- No attack can send an email without approval
- Normal task success did not drop by more than a few points
Stretch goals
- Red-team set running in CI
- promptfoo scan with extra plugins
- Sandboxed code execution tool with escape tests