Area 02 · RAG · Concept 11 of 20

Query Rewriting

Let the LLM rephrase or expand the user’s question before you search.

11–20 · Production levelPhase 2 · Weeks 7–10≈ 45 min

The idea

Users type short, vague or conversational questions: "and what about for contractors?" after a previous turn. Searching for that text alone finds nothing useful. Query rewriting uses an LLM to turn it into a standalone search query using the chat history.

Variants include expansion (add synonyms and related terms), decomposition (split a compound question into sub-questions, search each) and multi-query (generate 3–5 phrasings and merge results with RRF).

Rewriting adds an LLM call before retrieval, so use a small fast model and keep the prompt tight.

Why it matters

Follow-up questions are where chat-with-docs systems most often fail. Rewriting fixes most of them.

Words to know

  • Standalone questionA query that makes sense without the chat history.

Try it without code

Rewrite a follow-up

  1. Ask a chat app: "What is the annual leave policy?" then "and for part-timers?"
  2. Ask it to rewrite the second question so it makes sense alone.

Use it in code

Turn a follow-up into a standalone query plus alternatives.

python
import json, anthropic
client = anthropic.Anthropic()

def rewrite(history, question):
    prompt = f"""Chat so far:
{history}

New question: {question}
Return JSON: {{"standalone": "...", "alternatives": ["...", "..."]}}"""
    msg = client.messages.create(model="claude-haiku-5-5", max_tokens=200, temperature=0,
                                 messages=[{"role": "user", "content": prompt}])
    return json.loads(msg.content[0].text)

print(rewrite("User: What is the annual leave policy?\nAssistant: 25 days for full-time staff.",
              "and for part-timers?"))
PRACTICE · TICK IT AFTER THIS

Add rewriting to your RAG chat and test it on 10 follow-up questions.

Common mistakes

  • Rewriting a clear question into a worse one; skip rewriting for first turns.
  • Losing names and codes during rewriting.

Go deeper