The idea
BM25 scores documents by the query words they contain, giving more weight to rare words and less to very long documents. It is the algorithm behind classic search engines and Elasticsearch’s default ranking.
Embeddings capture meaning but can blur exact strings: error code E-4012, a product SKU, a person’s surname, an API name. Keyword search finds those reliably. It also works without any model.
In Postgres, full-text search (tsvector, ts_rank) is close to BM25 and good enough for many apps; extensions add true BM25. In Python, rank_bm25 is handy for experiments.
Many retrieval failures are exact-match failures. Keyword search is the cheap fix, and half of hybrid search.
Words to know
- TF-IDFTerm frequency × inverse document frequency; BM25 refines it.
- tsvectorPostgres’s indexed representation of text for full-text search.
Try it without code
Break semantic search
- Search your embedded docs for an exact error code or product ID.
- Note whether the right chunk ranks first.
- Try the same with Ctrl+F logic.
Use it in code
BM25 over chunks with rank_bm25.
# pip install rank_bm25
from rank_bm25 import BM25Okapi
docs = ["Error E-4012 means the token expired.",
"Tokens are refreshed every hour.",
"Contact support for billing questions."]
bm25 = BM25Okapi([d.lower().split() for d in docs])
scores = bm25.get_scores("what is e-4012".lower().split())
for d, s in sorted(zip(docs, scores), key=lambda x: -x[1]):
print(round(s, 2), d)Add a Postgres full-text column (tsvector with a GIN index) to your chunks table and write a keyword search function.
Common mistakes
- No stemming or lowercasing, so "Refunds" misses "refund".
- Dropping keyword search entirely once embeddings work.