Area 02 · RAG · Concept 18 of 20

Agentic RAG

An agent decides when to search, where, and how many times, instead of one fixed lookup.

11–20 · Production levelPhase 2 · Weeks 7–10≈ 1.5 hours

The idea

Classic RAG always runs one search before answering. Agentic RAG gives the model search as a tool. It can skip search for small talk, search twice with refined queries, pick between sources (docs, SQL database, web), and stop when it has enough.

This handles complex questions much better, especially those needing several lookups. The cost is more calls, more latency and less predictability, so add step limits and log every search.

It is the bridge between this area and AI Agents: the agent loop, tool calling and stop conditions all apply.

Why it matters

Real users ask questions that one lookup cannot answer. Agentic retrieval is how assistants handle them.

Picture it

Words to know

  • Retrieval toolA search function exposed to the model as a callable tool.

Try it without code

Plan the searches

  1. Write a question that needs two lookups (e.g. compare two policies).
  2. List the searches you would run, in order.
  3. That plan is what an agentic RAG system decides on its own.

Use it in code

Expose search as a tool and loop until the model answers.

python
import anthropic
client = anthropic.Anthropic()
TOOLS = [{"name": "search_docs", "description": "Search the company handbook. Returns top passages.",
          "input_schema": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}]

def search_docs(query):          # plug in your hybrid search here
    return "\n".join(f"- {c}" for c in ["Full-time: 25 days leave.", "Part-time: pro rata."])

messages = [{"role": "user", "content": "Compare leave for full-time and part-time staff."}]
for step in range(5):
    msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=800, tools=TOOLS, messages=messages)
    messages.append({"role": "assistant", "content": msg.content})
    if msg.stop_reason != "tool_use":
        print(msg.content[-1].text); break
    results = [{"type": "tool_result", "tool_use_id": b.id, "content": search_docs(**b.input)}
               for b in msg.content if b.type == "tool_use"]
    messages.append({"role": "user", "content": results})
PRACTICE · TICK IT AFTER THIS

Give your PDF chat a search tool and a step limit of 4, and log how many searches each question uses.

Common mistakes

  • No step limit.
  • Tool descriptions so vague the model never searches.

Go deeper