The idea
A model generates plausible text, and plausible is not the same as true. When it lacks the facts, it can produce fluent, confident, wrong answers: invented citations, wrong numbers, APIs that do not exist. This is a property of how generation works, not a rare bug.
The main defences are grounding (give it the source text and tell it to answer only from that), citations (require it to quote or point to the passage it used), and explicit permission to say it does not know. Lower temperature helps on factual tasks.
Then check. For high-stakes outputs, verify claims against the sources automatically (faithfulness evals), validate structured fields, and keep a human in the loop where errors are costly.
Trust is the product. One confident wrong answer in front of a customer can outweigh a hundred right ones.
Words to know
- GroundingTying answers to provided source material.
- FaithfulnessWhether every claim is supported by the given context.
Try it without code
Provoke and then prevent one
- Ask a chat app for "three academic papers about X by author Y" for an obscure pair.
- Check whether the papers exist.
- Ask again with: "If you are not certain a paper exists, say you don’t know."
Use it in code
A grounded prompt that requires quotes and allows "not in the document".
import anthropic
client = anthropic.Anthropic()
doc = open("policy.txt").read()
q = "What is the notice period for contractors?"
prompt = f"""<document>{doc}</document>
Answer the question using only the document.
First copy the exact supporting sentences into <quotes>.
If the document does not answer it, reply exactly: NOT IN DOCUMENT.
Question: {q}"""
msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=400, temperature=0,
messages=[{"role": "user", "content": prompt}])
print(msg.content[0].text)Write 10 questions about a document, including 3 it cannot answer. Count how often the model correctly says NOT IN DOCUMENT.
Common mistakes
- Trusting citations without checking the quoted text exists.
- Telling users the model is "always accurate".