RAG or Tool Calling? Stop Retrieving What You Could Query

The smell

A team embeds their orders table into a vector database so the agent can "search" it. A question arrives: how many orders shipped late last month? The system embeds the question, retrieves the twenty nearest order records, and asks the model to count.

The answer will be wrong, and it will be wrong in the worst way — confidently, plausibly, and without any signal that retrieval returned twenty of four thousand relevant rows.

A SQL query answers this exactly, in milliseconds, with no embedding pipeline to maintain. The team built a fuzzy, expensive approximation of `SELECT COUNT(*)`.

What each mechanism is actually for

Retrieval finds semantically similar text when you cannot express the query precisely. "What does our policy say about damaged returns?" has no exact-match formulation — the relevant paragraph may never use the word damaged. That is retrieval's job and nothing else does it well.

A tool executes a precise operation against structured data. Counts, filters, joins, aggregates, lookups by key. It returns exact results and it either succeeds or fails visibly.

The distinction is not document versus database. It is whether the question has a precise formulation. If you can write the query, write the query.

The decision rule

Ask one question: could a competent engineer with access to this data write an exact query that answers this?

If yes, expose a tool. Anything involving counting, summing, filtering by a known field, sorting, or looking up by identifier is a query, and running it through embeddings converts an exact answer into a probabilistic one for no benefit.

If no — the question is about meaning, the relevant passage cannot be located by field, the phrasing varies unpredictably — that is retrieval.

And when the answer is both, which is common, use both.

Could you write an exact query for this?
  │
  ├─ Yes ──► Tool. SQL, an API call, a key lookup.
  │           Counts, filters, joins, aggregates, IDs.
  │
  ├─ No ───► Retrieval. Policy text, docs, tickets, anything
  │           where the wording varies and meaning is the key.
  │
  └─ Both ─► Retrieval to find the relevant records,
              then a tool to compute over them exactly.

The hybrid is usually right

Most real questions have a structured part and a semantic part, and the strongest architecture uses each for what it is good at.

"Which enterprise customers complained about latency last quarter?" splits cleanly. Enterprise and last quarter are exact filters — a tool. Complained about latency is semantic — retrieval over ticket text. Run the filter first, then search within the filtered set.

Filtering before retrieving is the part teams get backwards. Searching the whole corpus and filtering the results afterwards means the semantic top-k is drawn from everything, so the enterprise tickets you needed may not survive into the twenty rows you kept. Filter first and the retrieval budget is spent entirely on candidates that qualify.

# Backwards: top-k comes from the whole corpus, then most of it is discarded
hits = vector.search(query, k=20)
hits = [h for h in hits if h.tier == "enterprise" and h.q == "Q2"]   # maybe 2 left

# Right: the filter runs first, so all 20 slots go to eligible candidates
ids  = db.query("SELECT id FROM tickets WHERE tier='enterprise' AND q='Q2'")
hits = vector.search(query, k=20, filter={"id": {"$in": ids}})

Why the wrong choice is hard to catch

A failed tool call is loud. The query errors, the endpoint 404s, something appears in a log.

Retrieval never fails. It always returns its k nearest neighbours, ranked, looking exactly as authoritative when they are irrelevant as when they are perfect. There is no error to alert on and no signal in the response that says these are the wrong twenty rows.

So an agent doing arithmetic over retrieved records produces confident wrong numbers indefinitely, and the only way to find out is for someone to check an answer by hand. That asymmetry — loud failure versus silent failure — is the strongest practical argument for using a tool whenever a tool would do.

What this looks like in an interview

Describe a RAG system and a good candidate asks what the data actually is. If it is a database, proposing embeddings for it is the answer that marks someone as having read about the pattern rather than operated it.

The points that land: retrieval is for semantic similarity over text, tools are for precise operations over structured data, and the tell is whether you could write the query. Then the hybrid, with the filter-before-search ordering. Then the failure asymmetry — that retrieval degrades silently while tools fail loudly, which is why you should prefer the tool wherever both would work.

That last one is the part most people miss, and it is the one that shows you have debugged a system where the numbers were quietly wrong.