Building AI Agents That Survive Production · Part 18 of 44

Hybrid Search and Reranking: The Two Upgrades Worth Their Latency

BM25 plus embeddings, then a cross-encoder over the top-k. The two stages buy different things and most teams cannot say which one they added. Hybrid search buys recall because lexical and vector retrieval fail on different queries; reranking buys precision because a cross-encoder reads the query and the document together. Here is what each one measurably adds, and how to tell when the reranker is buying you a rounding error.

Two retrievers, two different blind spots

A pure vector retriever misses queries it should have nailed, and it misses them in a specific way. Search for the error code ECONNRESET, or SKU 7731-B, or the function name retry_with_backoff, and an embedding model hands back documents that are about connection failures, inventory and retry logic. It will not reliably return the one document containing that exact token, because the token contributed a fraction of a 1,536-dimensional average and nothing in the architecture privileges it.

BM25 misses the opposite query. Ask why checkouts fail intermittently against a corpus that only ever says transaction timeouts under sustained load, and the lexical overlap is a stopword. No amount of term weighting recovers a document that shares no terms with the question.

That is the entire argument for hybrid search, and it is worth stating precisely, because the usual framing is wrong. You are not averaging two mediocre rankers into a better one. You are covering each retriever’s blind spot with a retriever that does not have it. The gain shows up as recall on the queries where one stage returns nothing useful — and in most production query logs that is a larger share of traffic than anyone estimated.

# Two retrievers over the SAME text, run independently.
# pip install rank-bm25

from rank_bm25 import BM25Okapi

def tokenize(s: str) -> list[str]:
    return re.findall(r"[a-z0-9_.-]+", s.lower())   # keep ECONNRESET, 7731-B, retry_with_backoff

# Built once at ingest, over the exact strings you embedded.
bm25 = BM25Okapi([tokenize(c["text"]) for c in chunks], k1=1.2, b=0.75)

def retrieve_lexical(query: str, n: int) -> list[str]:
    scores = bm25.get_scores(tokenize(query))
    top = sorted(range(len(scores)), key=lambda i: -scores[i])[:n]
    return [chunks[i]["id"] for i in top]

def retrieve_vector(query: str, n: int) -> list[str]:
    hits = index.query(vector=embed(query), top_k=n)
    return [h["id"] for h in hits]

BM25 is not a legacy baseline

BM25 is a scoring function, not a keyword filter, and the parameters inside it are why it has outlived every announcement of its death. Term frequency saturates, so the tenth occurrence of a word adds much less than the second — that is k1. Document length is normalised, so a long document does not win by containing everything — that is b. Rare terms are weighted up by inverse document frequency, which is exactly the behaviour you want for identifiers, stack traces and product codes.

Anthropic’s contextual retrieval write-up puts a number on the addition. Contextual embeddings alone cut their top-20 retrieval failure rate by 35%, from 5.7% to 3.7%. Running a contextual BM25 index alongside took the reduction to 49%, down to 2.9%. A third of the remaining failures, removed by a scoring function from 1994 sitting next to the vector store.

One implementation note that quietly breaks pipelines: run BM25 over the same text you embedded, including any context you prepended before embedding. Two indexes over different strings are two different corpora, and the fusion step will happily rank chunks that only exist in one of them.

Fuse ranks, not scores

The first instinct is to normalise both scores and take a weighted sum. It is a trap worth understanding rather than just avoiding. A BM25 score is unbounded and corpus-dependent — its scale moves with your IDF table, so the same document scores differently after the next ingest. Cosine similarity is bounded but badly calibrated; real distributions sit in a narrow band like 0.60 to 0.85, so min-max normalising a result set rescales noise differently on every query.

Reciprocal rank fusion sidesteps all of it by throwing the scores away and using only positions. For each document, sum 1/(k + rank) across the lists it appears in. The conventional k is 60, from the 2009 SIGIR paper by Cormack, Clarke and Buettcher that introduced the method, and it damps the head of each list so one retriever’s confident first place cannot dominate — while still rewarding documents that both retrievers liked.

RRF is also the honest default because it has no corpus-specific parameter to fit. If you later want to favour one retriever, add a weight per list rather than touching k. The weight is interpretable and k is not.

# Reciprocal rank fusion: positions only, no score normalisation.
# Cormack, Clarke & Buettcher, SIGIR 2009.

def rrf(ranked_lists: list[list[str]], k: int = 60,
        weights: list[float] | None = None) -> list[tuple[str, float]]:
    weights = weights or [1.0] * len(ranked_lists)
    fused: dict[str, float] = {}
    for w, ids in zip(weights, ranked_lists):
        for rank, doc_id in enumerate(ids, start=1):     # rank is 1-based
            fused[doc_id] = fused.get(doc_id, 0.0) + w / (k + rank)
    return sorted(fused.items(), key=lambda kv: -kv[1])

# Why not sum the scores: BM25 is unbounded and shifts with your IDF table,
# cosine sits in a narrow band like 0.60-0.85. Min-max normalising a result
# set rescales noise differently on every single query.

candidates = rrf([retrieve_lexical(q, 150), retrieve_vector(q, 150)])

A reranker is a different model class, not a better embedder

The model you retrieve with is a bi-encoder: it encodes the query and the document separately, into vectors that never meet until a dot product at the end. That separation is the whole reason retrieval is fast — document vectors are computed once at ingest and live in an ANN index, and the query only ever touches them through one cheap similarity.

A cross-encoder reranker gives up the separation. It takes the query and one document as a single input and runs full attention across both, so every query token can attend to every document token. The relevance judgment is computed with the question in hand, rather than compressed into a fixed vector months earlier. That is a strictly richer computation, and it is why rerankers reliably beat the retriever that fed them.

It is also why a reranker can never be the retriever. Nothing is precomputable: scoring N documents is N forward passes with the query attached. Over a million chunks that is not slow, it is impossible. The cross-encoder earns its accuracy by doing work that does not scale — which is exactly why it belongs in a second stage over a short list.

The funnel, and the one number that bounds it

A reranker cannot retrieve. It can only reorder what stage one handed it, so your ceiling is fixed before it runs: recall@N of the fused candidate list. If the right chunk is not among the 150 you fetched, no reranker will produce it, and an end-to-end metric will blame the wrong stage.

Anthropic’s configuration is a good reference shape — retrieve 150 chunks, rerank them, pass the top 20 to the model. Adding that stage moved their failure-rate reduction from 49% to 67%, 2.9% down to 1.9%. Another third of the remaining failures, from reordering alone.

Pick N by measuring stage-one recall, not by taste. Push N up until recall@N stops climbing, then stop — every candidate past that point is latency and per-document reranking cost buying nothing. Pick the final k by what generation can actually use without diluting attention, which is usually a much smaller number than the context window permits.

# The funnel. N is bounded by measured stage-1 recall; k by what
# generation can actually use.
import cohere

co = cohere.ClientV2()
N_CANDIDATES, K_FINAL = 150, 20

def search(query: str) -> list[dict]:
    fused = rrf([retrieve_lexical(query, N_CANDIDATES),
                 retrieve_vector(query, N_CANDIDATES)])[:N_CANDIDATES]
    docs = [by_id[doc_id] for doc_id, _ in fused]

    res = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=[d["text"] for d in docs],
        top_n=K_FINAL,
    )
    # relevance_score is normalised to [0,1] and IS comparable across
    # documents for one query - unlike the two scores you just fused.
    return [dict(docs[r.index], score=r.relevance_score) for r in res.results]

Measure the reranker before you ship it

There are two questions here and they need separate numbers. Did stage one find the answer, and did stage two put it near the top? Recall@N answers the first. NDCG@k or success@k over the reranked list answers the second. One blended pipeline metric hides which half is broken, which is how teams end up tuning a reranker to fix a retrieval bug.

The diagnostic that saves the most time is a two-line comparison: recall@k of the fused list against recall@k after reranking, at the same k. If they are close, the reranker has nothing to fix — your candidates were already ordered well, and the hundreds of milliseconds you are about to add buy a rounding error. If recall@N is high and recall@k is low, reranking is precisely the missing piece.

Fifty labelled query-document pairs from your own traffic will tell you more than any public leaderboard. BEIR and friends measure general web and academic retrieval; your corpus is neither, and reranker gains are strongly corpus-dependent.

# Two questions, two measurements. Never report one number.

def recall_at(ranked_ids: list[str], relevant: set[str], k: int) -> float:
    return len(set(ranked_ids[:k]) & relevant) / max(len(relevant), 1)

def audit(labelled, n=150, k=20):
    ceiling = stage1 = reranked = 0.0
    for q, relevant in labelled:
        fused = [i for i, _ in rrf([retrieve_lexical(q, n), retrieve_vector(q, n)])]
        ceiling  += recall_at(fused, relevant, n)   # what stage 2 can never exceed
        stage1   += recall_at(fused, relevant, k)   # fused list, truncated
        reranked += recall_at([d["id"] for d in search(q)], relevant, k)
    m = len(labelled)
    print(f"recall@{n} (ceiling) {ceiling/m:.3f}")
    print(f"recall@{k} fused     {stage1/m:.3f}")
    print(f"recall@{k} reranked  {reranked/m:.3f}")

# stage1 ~= reranked  -> the reranker is buying a rounding error. Drop it.
# ceiling low         -> fix stage 1; the reranker is not your problem.
# ceiling high, stage1 low -> reranking is exactly the missing piece.

When to skip the reranker

Skip it when the query distribution is dominated by exact lookups. If most traffic is an identifier or an error string, BM25 already put the right document first and the cross-encoder is re-deciding a question that was never close.

Skip it when the latency budget is genuinely tight. A hosted reranker is a network round trip plus a forward pass per candidate, and reported latencies for the hosted APIs land in the hundreds of milliseconds, scaling with how many candidates you send. Inside an interactive agent loop that is a visible pause, and the right fix is often a smaller N rather than no reranker at all.

What is not a reason to skip it is a large context window. The tempting move is to drop the reranker and pass all 150 chunks, and it is usually the worse trade: you pay for all of them on every query, and position inside the prompt still affects what the model actually uses. Reranking to 20 is cheaper per query, and the ordering it produces is doing work even when everything would have fit.

The interview version of this is two sentences. Hybrid search buys recall, because lexical and vector retrieval fail on different queries. Reranking buys precision, because a cross-encoder reads the query and the document together — and it buys it only over a candidate list whose recall you have already measured.

PRACTISE IT · SYSTEM DESIGN LABS

Work the same ideas as a full interview case study, with pressure questions and model answers.

Keep reading

Related articles