Area 06 · AI Security · Concept 16 of 20

Data Poisoning

Bad data planted in training sets or in your RAG documents to change the model’s behaviour.

11–20 · Production levelPhase 7 · Weeks 24–26≈ 45 min

The idea

If attackers can influence what a model learns from or retrieves, they can change its behaviour. In fine-tuning, a few hundred crafted examples can plant a backdoor triggered by a phrase. In RAG, one edited wiki page or uploaded document can steer answers or carry injections for every user who retrieves it.

Defences: control who can add or edit indexed content, record provenance (who, when, source) for every chunk, review changes to high-impact sources, scan new documents for injection patterns, and monitor for sudden answer changes on your golden set.

For fine-tuning, use trusted data sources, deduplicate, sample and review, and keep a held-out eval to catch behaviour shifts.

Why it matters

Your knowledge base is now an attack surface. Anyone who can write to it can influence your AI.

Words to know

  • BackdoorHidden behaviour triggered by a specific input.
  • ProvenanceRecord of where data came from and who changed it.

Try it without code

Map write access

  1. List every source your RAG indexes.
  2. Write who can edit each one.
  3. Mark sources open to customers or the public.

Use it in code

Provenance and an injection scan at ingestion time.

python
import re, hashlib, datetime

SUSPICIOUS = [r"ignore (all )?(previous|prior) instructions", r"you are now", r"system prompt",
              r"(send|forward|email) .* to [\w.+-]+@", r"<\s*script"]

def ingest_check(text, source, author):
    hits = [p for p in SUSPICIOUS if re.search(p, text, re.I)]
    meta = {"source": source, "author": author,
            "sha256": hashlib.sha256(text.encode()).hexdigest(),
            "ingested_at": datetime.datetime.utcnow().isoformat(),
            "quarantined": bool(hits), "flags": hits}
    return meta     # store with every chunk; quarantined docs need review before indexing
PRACTICE · TICK IT AFTER THIS

Add provenance metadata and a quarantine step to your RAG ingestion, and test it with a poisoned document.

Common mistakes

  • Indexing user uploads into a shared index without review.
  • No way to trace an answer back to who wrote the source.

Go deeper