The idea
Public-facing AI apps receive abuse, harassment, self-harm disclosures, sexual content and illegal requests, and can produce harmful text if jailbroken. Moderation classifies content into categories and lets you block, warn, route to a human or show support resources.
Use a moderation model or API on both inputs and outputs, decide a policy per category (block, allow, escalate), and log decisions for review. Some categories need care, not just blocking: for self-harm, show helpful resources.
Tune for your audience: a children’s education app needs stricter settings than a security research tool.
It protects users, your brand and your compliance position, and is often required by app stores and enterprise customers.
Words to know
- PolicyWhat you do for each content category and severity.
Try it without code
Write a policy table
- List 6 content categories.
- For each, write: block, allow, warn or escalate, and what the user sees.
Use it in code
Moderate input and output with a moderation endpoint.
# pip install openai
from openai import OpenAI
client = OpenAI()
POLICY = {"self-harm": "support", "harassment": "block", "violence": "block", "sexual/minors": "block"}
def moderate(text):
res = client.moderations.create(model="omni-moderation-latest", input=text).results[0]
flagged = [c for c, hit in res.categories.model_dump(by_alias=True).items() if hit]
actions = {POLICY.get(c, "review") for c in flagged}
if "block" in actions: return "block", flagged
if "support" in actions: return "support", flagged
return ("review" if flagged else "allow"), flaggedModerate both directions in your chatbot and write the user-facing message for each action.
Common mistakes
- Moderating input only.
- A single "blocked" message for a user in distress.