The idea
Redis is an in-memory data store with sub-millisecond reads. As a cache, it holds results that are expensive to compute and read often: user profiles, permission lookups, embedding results, and exact-match LLM responses for identical requests.
The common pattern is cache-aside: check Redis; on a miss, compute, store with a TTL, return. Choose keys carefully (include tenant, model and prompt version in LLM cache keys) and decide how entries are invalidated when underlying data changes.
Redis also powers rate limiters, queues, sessions, locks and pub/sub, so it is usually already in an AI stack.
Caching cuts latency and cost at the same time, and Redis is the default tool for it.
Words to know
- TTLTime to live: when a key expires.
- Cache-asideThe app checks the cache and fills it on a miss.
Try it without code
Try Redis
- Run: docker run -p 6379:6379 redis:7
- Run redis-cli, then SET greeting hello EX 10, GET greeting, wait 10 seconds and GET again.
Use it in code
Cache-aside for exact-match LLM responses and embeddings.
import hashlib, json, redis
r = redis.Redis(decode_responses=True)
def cache_key(kind, payload):
return f"{kind}:" + hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()
def cached_completion(call_llm, model, prompt_version, messages, ttl=3600):
key = cache_key("llm", {"m": model, "v": prompt_version, "msgs": messages})
if (hit := r.get(key)) is not None:
return json.loads(hit)
result = call_llm(model, messages)
r.set(key, json.dumps(result), ex=ttl)
return resultCache embeddings of repeated queries in your RAG app and report hit rate and latency saved over 200 requests.
Common mistakes
- Cache keys missing the tenant or prompt version.
- No TTL, so stale data lives forever.