The idea
Users type short, vague or conversational questions: "and what about for contractors?" after a previous turn. Searching for that text alone finds nothing useful. Query rewriting uses an LLM to turn it into a standalone search query using the chat history.
Variants include expansion (add synonyms and related terms), decomposition (split a compound question into sub-questions, search each) and multi-query (generate 3–5 phrasings and merge results with RRF).
Rewriting adds an LLM call before retrieval, so use a small fast model and keep the prompt tight.
Follow-up questions are where chat-with-docs systems most often fail. Rewriting fixes most of them.
Words to know
- Standalone questionA query that makes sense without the chat history.
Try it without code
Rewrite a follow-up
- Ask a chat app: "What is the annual leave policy?" then "and for part-timers?"
- Ask it to rewrite the second question so it makes sense alone.
Use it in code
Turn a follow-up into a standalone query plus alternatives.
import json, anthropic
client = anthropic.Anthropic()
def rewrite(history, question):
prompt = f"""Chat so far:
{history}
New question: {question}
Return JSON: {{"standalone": "...", "alternatives": ["...", "..."]}}"""
msg = client.messages.create(model="claude-haiku-5-5", max_tokens=200, temperature=0,
messages=[{"role": "user", "content": prompt}])
return json.loads(msg.content[0].text)
print(rewrite("User: What is the annual leave policy?\nAssistant: 25 days for full-time staff.",
"and for part-timers?"))Add rewriting to your RAG chat and test it on 10 follow-up questions.
Common mistakes
- Rewriting a clear question into a worse one; skip rewriting for first turns.
- Losing names and codes during rewriting.