The idea
Classic RAG always runs one search before answering. Agentic RAG gives the model search as a tool. It can skip search for small talk, search twice with refined queries, pick between sources (docs, SQL database, web), and stop when it has enough.
This handles complex questions much better, especially those needing several lookups. The cost is more calls, more latency and less predictability, so add step limits and log every search.
It is the bridge between this area and AI Agents: the agent loop, tool calling and stop conditions all apply.
Real users ask questions that one lookup cannot answer. Agentic retrieval is how assistants handle them.
Picture it
Words to know
- Retrieval toolA search function exposed to the model as a callable tool.
Try it without code
Plan the searches
- Write a question that needs two lookups (e.g. compare two policies).
- List the searches you would run, in order.
- That plan is what an agentic RAG system decides on its own.
Use it in code
Expose search as a tool and loop until the model answers.
import anthropic
client = anthropic.Anthropic()
TOOLS = [{"name": "search_docs", "description": "Search the company handbook. Returns top passages.",
"input_schema": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}]
def search_docs(query): # plug in your hybrid search here
return "\n".join(f"- {c}" for c in ["Full-time: 25 days leave.", "Part-time: pro rata."])
messages = [{"role": "user", "content": "Compare leave for full-time and part-time staff."}]
for step in range(5):
msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=800, tools=TOOLS, messages=messages)
messages.append({"role": "assistant", "content": msg.content})
if msg.stop_reason != "tool_use":
print(msg.content[-1].text); break
results = [{"type": "tool_result", "tool_use_id": b.id, "content": search_docs(**b.input)}
for b in msg.content if b.type == "tool_use"]
messages.append({"role": "user", "content": results})Give your PDF chat a search tool and a step limit of 4, and log how many searches each question uses.
Common mistakes
- No step limit.
- Tool descriptions so vague the model never searches.