The idea
Embedding search compares a query vector with chunk vectors computed separately. A reranker (a cross-encoder) reads the query and one chunk together and outputs a relevance score. That joint reading is far more accurate, but too slow to run on every chunk.
So retrieval becomes two stages: a fast search returns 20–100 candidates, the reranker scores each one, and only the top 3–8 go into the prompt. This raises precision and lets you send fewer tokens.
Rerankers are available as open models (BGE reranker, mixedbread) and APIs (Cohere, Voyage). Measure the gain on your eval set; it is usually large.
Better chunks in the prompt means better answers and lower cost. Reranking is often the best quality-per-euro step in RAG.
Words to know
- Bi-encoderEncodes query and document separately (embeddings).
- Cross-encoderReads query and document together to score relevance.
Try it without code
Rerank in your head
- Take the top 10 chunks for a question.
- Order them by how well they actually answer it.
- Compare with the vector order.
Use it in code
Rerank candidates with an open cross-encoder.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-base")
question = "How long do refunds take?"
candidates = ["Refunds are processed within 5 business days.",
"We accept Visa and Mastercard.",
"Refund requests must be made within 30 days of purchase."]
scores = reranker.predict([(question, c) for c in candidates])
for s, c in sorted(zip(scores, candidates), reverse=True):
print(round(float(s), 3), c)Add reranking after hybrid search and report Recall@5 and answer quality before and after.
Common mistakes
- Reranking only 5 candidates; it can only reorder what it gets.
- Ignoring reranker latency in your budget.