Area 05 · AI System Design · Concept 03 of 20

Semantic Caching

Return a saved answer when a new question means the same thing as an old one.

01–10 · The basePhase 6 · Weeks 21–23≈ 1 hour

The idea

Many users ask the same things in different words. An exact-match cache misses "how do I reset my password" vs "forgot password, what now?". A semantic cache embeds each question, looks for a stored question above a similarity threshold, and returns its saved answer instead of calling the model.

Hits are much cheaper and faster than a model call. The risk is wrong hits: two questions that look similar but need different answers ("cancel order" vs "cancel subscription"). Tune the threshold on labelled pairs and scope the cache (per tenant, per language, per prompt version).

Expire entries when underlying data changes, and never cache personalised answers across users.

Why it matters

For FAQ-style traffic, semantic caching can remove a large share of model calls.

Words to know

  • Hit rateShare of requests served from cache.
  • False hitA cached answer returned for a question that needed a different one.

Try it without code

Find duplicates

  1. Take 50 support questions.
  2. Group the ones that should share one answer.
  3. Estimate the possible hit rate.

Use it in code

An in-memory semantic cache with a threshold.

python
import numpy as np
from sentence_transformers import SentenceTransformer

emb = SentenceTransformer("BAAI/bge-small-en-v1.5")

class SemanticCache:
    def __init__(self, threshold=0.92):
        self.keys, self.values, self.t = [], [], threshold
    def get(self, q):
        if not self.keys: return None
        v = emb.encode(q, normalize_embeddings=True)
        sims = np.array(self.keys) @ v
        i = int(sims.argmax())
        return self.values[i] if sims[i] >= self.t else None
    def put(self, q, answer):
        self.keys.append(emb.encode(q, normalize_embeddings=True)); self.values.append(answer)

cache = SemanticCache()
cache.put("How do I reset my password?", "Go to Settings > Security > Reset.")
print(cache.get("forgot my password, what now?"))
PRACTICE · TICK IT AFTER THIS

Pick a threshold using 30 labelled same/different question pairs, then measure hit rate on 200 questions.

Common mistakes

  • One global cache across tenants.
  • Caching answers that depend on the user or on live data.

Go deeper