Area 02 · RAG · Concept 10 of 20

Reranking

A second model re-scores the top results so the best chunks go into the prompt.

01–10 · The basePhase 2 · Weeks 7–10≈ 1 hour

The idea

Embedding search compares a query vector with chunk vectors computed separately. A reranker (a cross-encoder) reads the query and one chunk together and outputs a relevance score. That joint reading is far more accurate, but too slow to run on every chunk.

So retrieval becomes two stages: a fast search returns 20–100 candidates, the reranker scores each one, and only the top 3–8 go into the prompt. This raises precision and lets you send fewer tokens.

Rerankers are available as open models (BGE reranker, mixedbread) and APIs (Cohere, Voyage). Measure the gain on your eval set; it is usually large.

Why it matters

Better chunks in the prompt means better answers and lower cost. Reranking is often the best quality-per-euro step in RAG.

Words to know

  • Bi-encoderEncodes query and document separately (embeddings).
  • Cross-encoderReads query and document together to score relevance.

Try it without code

Rerank in your head

  1. Take the top 10 chunks for a question.
  2. Order them by how well they actually answer it.
  3. Compare with the vector order.

Use it in code

Rerank candidates with an open cross-encoder.

python
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")
question = "How long do refunds take?"
candidates = ["Refunds are processed within 5 business days.",
              "We accept Visa and Mastercard.",
              "Refund requests must be made within 30 days of purchase."]
scores = reranker.predict([(question, c) for c in candidates])
for s, c in sorted(zip(scores, candidates), reverse=True):
    print(round(float(s), 3), c)
PRACTICE · TICK IT AFTER THIS

Add reranking after hybrid search and report Recall@5 and answer quality before and after.

Common mistakes

  • Reranking only 5 candidates; it can only reorder what it gets.
  • Ignoring reranker latency in your budget.

Go deeper