The idea
If attackers can influence what a model learns from or retrieves, they can change its behaviour. In fine-tuning, a few hundred crafted examples can plant a backdoor triggered by a phrase. In RAG, one edited wiki page or uploaded document can steer answers or carry injections for every user who retrieves it.
Defences: control who can add or edit indexed content, record provenance (who, when, source) for every chunk, review changes to high-impact sources, scan new documents for injection patterns, and monitor for sudden answer changes on your golden set.
For fine-tuning, use trusted data sources, deduplicate, sample and review, and keep a held-out eval to catch behaviour shifts.
Your knowledge base is now an attack surface. Anyone who can write to it can influence your AI.
Words to know
- BackdoorHidden behaviour triggered by a specific input.
- ProvenanceRecord of where data came from and who changed it.
Try it without code
Map write access
- List every source your RAG indexes.
- Write who can edit each one.
- Mark sources open to customers or the public.
Use it in code
Provenance and an injection scan at ingestion time.
import re, hashlib, datetime
SUSPICIOUS = [r"ignore (all )?(previous|prior) instructions", r"you are now", r"system prompt",
r"(send|forward|email) .* to [\w.+-]+@", r"<\s*script"]
def ingest_check(text, source, author):
hits = [p for p in SUSPICIOUS if re.search(p, text, re.I)]
meta = {"source": source, "author": author,
"sha256": hashlib.sha256(text.encode()).hexdigest(),
"ingested_at": datetime.datetime.utcnow().isoformat(),
"quarantined": bool(hits), "flags": hits}
return meta # store with every chunk; quarantined docs need review before indexingAdd provenance metadata and a quarantine step to your RAG ingestion, and test it with a poisoned document.
Common mistakes
- Indexing user uploads into a shared index without review.
- No way to trace an answer back to who wrote the source.