Area 06 · AI Security · Concept 08 of 20

Data Leakage

The model revealing your system prompt, other users’ data or secrets.

01–10 · The basePhase 7 · Weeks 24–26≈ 45 min

The idea

Anything in the context can come out in the output. That includes your system prompt, retrieved documents the user should not see, other users’ data if it was mixed in, API keys placed in prompts, and memory from other sessions.

The rule: do not put anything in the context that this user is not allowed to see. Prompts will be extracted eventually, so keep them free of secrets and treat them as public. Enforce access control before retrieval, not after generation.

Watch for leakage paths through tools (an agent reading another user’s file), caches (shared semantic cache) and logs (traces visible to too many staff).

Why it matters

Leaking one customer’s data to another is a reportable breach. The model will not keep secrets for you.

Words to know

  • System prompt leakageThe model revealing its hidden instructions.

Try it without code

Context audit

  1. Dump the full context of one request.
  2. Mark anything the current user should not see.
  3. If anything is marked, you have a leak waiting to happen.

Use it in code

An output check for canary strings and known secrets.

python
import os, re

CANARY = "CANARY-7f3a"      # put this in your system prompt; it should never appear in output
SECRETS = [os.environ.get("STRIPE_KEY", ""), os.environ.get("DB_PASSWORD", "")]

def leaks(output: str) -> list[str]:
    found = []
    if CANARY in output:
        found.append("system prompt")
    found += ["secret" for s in SECRETS if s and s in output]
    if re.search(r"tenant_id\s*[:=]", output):
        found.append("internal identifiers")
    return found
PRACTICE · TICK IT AFTER THIS

Add a canary to your system prompt, run your red-team set, and alert whenever the canary appears in output.

Common mistakes

  • Secrets in prompts "because the model needs them".
  • Fetching all documents then asking the model to only use allowed ones.

Go deeper