Area 06 · AI Security · Build this

Red-team and harden your agent

Attack the research agent from Phase 3 with 20 injection and jailbreak prompts, record what gets through, then add layered defences and prove the attack success rate drops without breaking normal use.

Phase 7 · Weeks 24–26

What you will have at the end

  • redteam.jsonl with 20+ attacks, goals and expected safe behaviour
  • A before/after report of attack success rate and false refusals
  • Guardrails, output validation, tool permissions and an approval step in code

Step by step

  1. Threat modelFill in the OWASP LLM Top 10 table for your agent and list 5 attacker goals.
  2. Write the attacks20 attacks across goals: direct injection, indirect injection in a fetched page, prompt extraction, jailbreak, data exfiltration via email tool.
  3. Baseline runRun all attacks, judge success, and record the attack success rate plus 30 normal tasks as a quality baseline.
  4. Input guardrailsLength and secret checks, an injection classifier, and untrusted-content wrapping for fetched pages.
  5. Least privilegeNarrow tools, read-only by default, identity from the session, and drop outbound tools after untrusted content is read.
  6. Output validation and approvalValidated final output, canary check for prompt leakage, and human approval for send_email.
  7. Audit logLog every tool call, block and approval as structured events.
  8. Re-run and reportRun attacks and normal tasks again. Report attack success rate, false refusals and added latency in SECURITY.md.
DONE WHEN
  • Attack success rate is documented before and after
  • No attack can send an email without approval
  • Normal task success did not drop by more than a few points

Stretch goals

  • Red-team set running in CI
  • promptfoo scan with extra plugins
  • Sandboxed code execution tool with escape tests