Area 06 · AI Security · Concept 15 of 20

Content Moderation

Detect and block harmful content going in and coming out.

11–20 · Production levelPhase 7 · Weeks 24–26≈ 45 min

The idea

Public-facing AI apps receive abuse, harassment, self-harm disclosures, sexual content and illegal requests, and can produce harmful text if jailbroken. Moderation classifies content into categories and lets you block, warn, route to a human or show support resources.

Use a moderation model or API on both inputs and outputs, decide a policy per category (block, allow, escalate), and log decisions for review. Some categories need care, not just blocking: for self-harm, show helpful resources.

Tune for your audience: a children’s education app needs stricter settings than a security research tool.

Why it matters

It protects users, your brand and your compliance position, and is often required by app stores and enterprise customers.

Words to know

  • PolicyWhat you do for each content category and severity.

Try it without code

Write a policy table

  1. List 6 content categories.
  2. For each, write: block, allow, warn or escalate, and what the user sees.

Use it in code

Moderate input and output with a moderation endpoint.

python
# pip install openai
from openai import OpenAI
client = OpenAI()

POLICY = {"self-harm": "support", "harassment": "block", "violence": "block", "sexual/minors": "block"}

def moderate(text):
    res = client.moderations.create(model="omni-moderation-latest", input=text).results[0]
    flagged = [c for c, hit in res.categories.model_dump(by_alias=True).items() if hit]
    actions = {POLICY.get(c, "review") for c in flagged}
    if "block" in actions: return "block", flagged
    if "support" in actions: return "support", flagged
    return ("review" if flagged else "allow"), flagged
PRACTICE · TICK IT AFTER THIS

Moderate both directions in your chatbot and write the user-facing message for each action.

Common mistakes

  • Moderating input only.
  • A single "blocked" message for a user in distress.

Go deeper