The idea
Model output is untrusted input to the rest of your system. If you render it as HTML, it can carry script injection. If you run it as SQL or shell, it can drop tables. If you pass it to a tool, the arguments can be malicious. OWASP calls this improper output handling.
Validate by use: schema validation for structured data (Pydantic), allow-lists for values (status must be one of three), escaping for HTML and Markdown, parameterised queries for SQL, and never eval or exec model output outside a sandbox. Check links and emails in output against allowed domains.
For text shown to users, add content checks for things your policy forbids, like competitor mentions or personal data.
Many "AI vulnerabilities" are ordinary web vulnerabilities reached through model output.
Words to know
- Allow-listOnly explicitly permitted values pass.
Try it without code
Find the sinks
- List every place model output goes: UI, database, tools, emails, shell.
- Write the validation each one needs.
Use it in code
Validate structured output and sanitise text before rendering.
from typing import Literal
from pydantic import BaseModel, HttpUrl, field_validator
import bleach
ALLOWED_DOMAINS = {"docs.example.com", "help.example.com"}
class Action(BaseModel):
action: Literal["refund", "escalate", "reply"]
amount_eur: float = 0
link: HttpUrl | None = None
@field_validator("amount_eur")
@classmethod
def cap(cls, v):
if not 0 <= v <= 100: raise ValueError("amount out of range")
return v
@field_validator("link")
@classmethod
def domain(cls, v):
if v and v.host not in ALLOWED_DOMAINS: raise ValueError("link not allowed")
return v
def safe_html(model_text):
return bleach.clean(model_text, tags=["p", "b", "i", "ul", "li", "code"], strip=True)Make your agent’s final output a validated Pydantic object and add a test where the model returns a disallowed link.
Common mistakes
- Rendering model Markdown with raw HTML enabled.
- Building SQL strings from model output.