The idea
Many users ask the same things in different words. An exact-match cache misses "how do I reset my password" vs "forgot password, what now?". A semantic cache embeds each question, looks for a stored question above a similarity threshold, and returns its saved answer instead of calling the model.
Hits are much cheaper and faster than a model call. The risk is wrong hits: two questions that look similar but need different answers ("cancel order" vs "cancel subscription"). Tune the threshold on labelled pairs and scope the cache (per tenant, per language, per prompt version).
Expire entries when underlying data changes, and never cache personalised answers across users.
For FAQ-style traffic, semantic caching can remove a large share of model calls.
Words to know
- Hit rateShare of requests served from cache.
- False hitA cached answer returned for a question that needed a different one.
Try it without code
Find duplicates
- Take 50 support questions.
- Group the ones that should share one answer.
- Estimate the possible hit rate.
Use it in code
An in-memory semantic cache with a threshold.
import numpy as np
from sentence_transformers import SentenceTransformer
emb = SentenceTransformer("BAAI/bge-small-en-v1.5")
class SemanticCache:
def __init__(self, threshold=0.92):
self.keys, self.values, self.t = [], [], threshold
def get(self, q):
if not self.keys: return None
v = emb.encode(q, normalize_embeddings=True)
sims = np.array(self.keys) @ v
i = int(sims.argmax())
return self.values[i] if sims[i] >= self.t else None
def put(self, q, answer):
self.keys.append(emb.encode(q, normalize_embeddings=True)); self.values.append(answer)
cache = SemanticCache()
cache.put("How do I reset my password?", "Go to Settings > Security > Reset.")
print(cache.get("forgot my password, what now?"))Pick a threshold using 30 labelled same/different question pairs, then measure hit rate on 200 questions.
Common mistakes
- One global cache across tenants.
- Caching answers that depend on the user or on live data.