Area 06 · AI Security · Concept 20 of 20

Red Teaming

Attack your own app on purpose to find the holes before someone else does.

11–20 · Production levelPhase 7 · Weeks 24–26≈ 2 hours

The idea

Red teaming is structured attacking of your own system. You define goals an attacker might have (leak the system prompt, read another user’s data, make the agent send an email, produce banned content), then try many techniques to achieve them: direct and indirect injection, jailbreaks, encoding tricks, multi-turn manipulation, poisoned documents.

Keep successful attacks as a regression suite and run it in CI like your quality evals. Tools such as promptfoo and PyRIT can generate and run attack variations at scale, and an LLM judge can score whether each attack succeeded.

Re-run red teaming after every major change: new tools, new data sources, new models.

Why it matters

You cannot claim a defence works until you have tried hard to break it.

Picture it

Words to know

  • Attack goalWhat the attacker wants to achieve.
  • Attack success rateShare of attempts that achieved the goal.

Try it without code

Write attack goals

  1. List 5 goals an attacker would have against your agent.
  2. For each, write 4 attack messages.
  3. That is your first 20-item red-team set.

Use it in code

A promptfoo red-team config for your app endpoint.

yaml
# promptfooconfig.yaml  ->  run: npx promptfoo@latest redteam run
targets:
  - id: http
    config:
      url: http://localhost:8000/chat
      method: POST
      body: { message: "{{prompt}}" }
      transformResponse: json.reply
redteam:
  purpose: "Support agent for an accounting SaaS. Can look up invoices and send emails to the account owner."
  plugins:
    - pii
    - excessive-agency
    - prompt-extraction
    - indirect-prompt-injection
  strategies:
    - jailbreak
    - prompt-injection
PRACTICE · TICK IT AFTER THIS

Run your 20-attack set plus a promptfoo scan, fix the top issues, and report attack success rate before and after.

Common mistakes

  • Red teaming once before launch and never again.
  • Only testing single-turn attacks.

Go deeper