The idea
Anything in the context can come out in the output. That includes your system prompt, retrieved documents the user should not see, other users’ data if it was mixed in, API keys placed in prompts, and memory from other sessions.
The rule: do not put anything in the context that this user is not allowed to see. Prompts will be extracted eventually, so keep them free of secrets and treat them as public. Enforce access control before retrieval, not after generation.
Watch for leakage paths through tools (an agent reading another user’s file), caches (shared semantic cache) and logs (traces visible to too many staff).
Leaking one customer’s data to another is a reportable breach. The model will not keep secrets for you.
Words to know
- System prompt leakageThe model revealing its hidden instructions.
Try it without code
Context audit
- Dump the full context of one request.
- Mark anything the current user should not see.
- If anything is marked, you have a leak waiting to happen.
Use it in code
An output check for canary strings and known secrets.
import os, re
CANARY = "CANARY-7f3a" # put this in your system prompt; it should never appear in output
SECRETS = [os.environ.get("STRIPE_KEY", ""), os.environ.get("DB_PASSWORD", "")]
def leaks(output: str) -> list[str]:
found = []
if CANARY in output:
found.append("system prompt")
found += ["secret" for s in SECRETS if s and s in output]
if re.search(r"tenant_id\s*[:=]", output):
found.append("internal identifiers")
return foundAdd a canary to your system prompt, run your red-team set, and alert whenever the canary appears in output.
Common mistakes
- Secrets in prompts "because the model needs them".
- Fetching all documents then asking the model to only use allowed ones.