Semantic caching with Redis or Valkey
A semantic cache that stops repeat questions from reaching the model, on Redis or Valkey that runs in OSStend's cloud, your cloud account or your data center.
45 minutes with an engineer: we review your workload and sketch the design.
A semantic cache stores model responses and returns a stored answer when a new prompt is close enough in meaning to an earlier one. It skips the model call on a hit. Redis or Valkey suit it because they combine in-memory reads, expiry and vector search.
- PostgreSQL
- MySQL
- MongoDB
- Redis
- Valkey
- OpenSearch
- Kafka
- RabbitMQ
- AWS
- Google Cloud
- Azure
- Kubernetes
- OpenShift
- Prometheus
- Grafana
- Datadog
- Dynatrace
- Splunk
Why exact-match caching misses repeat questions
Users rarely repeat a question word for word, so an exact-match cache rarely hits. A semantic cache compares meaning instead, which raises the hit rate and introduces a new risk: returning a stored answer to a question that only looks similar.
- Paraphrases miss.
- Different wording of the same question produces a different cache key.
- Cost and latency repeat.
- Every miss is a full model call, billed and timed again.
- A loose match is a wrong answer.
- Two questions can look alike and need different answers, such as the same request about two different accounts.
- Stored answers go stale.
- If the source data changes, a cached answer keeps repeating the old fact until it expires.
- Cached answers can cross a boundary.
- A response generated for one user must not be served to a user who may not see that data.
How a semantic cache works
The application embeds each incoming prompt, searches a vector index of earlier prompts for the nearest one, and compares the distance to a threshold. Under the threshold it returns the stored response. Over it, it calls the model and stores the new pair.
Embed the prompt
Turn the incoming prompt into a vector with the same embedding model used for stored prompts.
Search the cache
Run a nearest-neighbour search over cached prompt vectors, filtered by tenant or permission scope.
Apply the threshold
If the closest distance is within the threshold, return the stored response. This is a hit.
Call the model on a miss
Otherwise call the model as usual, return the answer and store the prompt vector with the response and an expiry.
| Component | Role | Documentation |
|---|---|---|
| Redis or Valkey key-value store | Holds responses, with expiry and a memory limit | Redis LangCache describes configurable similarity thresholds, TTLs and eviction policies |
| Vector index | Finds the nearest cached prompt | Valkey Search is a module with HNSW approximate search and exact KNN |
| Filter fields | Scope hits to a tenant, model or language | Valkey Search supports tag, numeric and full-text filters alongside vector queries |
| Embedding model | Produces prompt vectors | Runs outside the cache; the model must be the same for stored and incoming prompts |
Sources: Redis and Valkey documentation, checked 30 Oct 2026. Redis describes LangCache as in preview on that date, so check current status.
Thresholds, eviction and cost
Three settings decide whether a semantic cache pays off: the similarity threshold, the expiry and eviction policy, and the hit rate they produce. The cost maths is a comparison of saved model calls against the cost of embedding every prompt and running the cache.
| Setting or driver | What it changes | How to approach it |
|---|---|---|
| Similarity threshold | Hit rate against wrong-answer risk | Start strict. Test with real prompt pairs, including near-misses that need different answers, and loosen only while those still miss. |
| Expiry (TTL) | Staleness | Set by how fast the underlying facts change. Invalidate on source changes where possible. |
| Eviction policy and memory limit | Which entries survive when memory is full | Set a memory limit and an eviction policy that matches your access pattern. Cache entries can always be rebuilt. |
| Persistence | Cold-start behaviour | Optional for a cache. Persisting avoids an empty cache after a restart, at some write cost. |
| Scope filters | Safety across users and tenants | Include tenant, permission group and model version in the lookup. |
| Cost maths | Net saving | Saving = hit rate × requests × cost of a model call, minus the embedding cost for every request and the cost of running the cache. Measure hit rate before committing. |
Cache settings follow the engine documentation: Redis LangCache and Valkey Search, checked 30 Oct 2026.
Keeping cached answers fresh
A cached answer is only as current as the data behind it. Use a short expiry for fast-changing facts, and where the source is a database, publish its changes so affected entries can be removed.
A change stream from the source database into Kafka can drive invalidation as well as re-embedding (see keeping embeddings fresh with Kafka and CDC), so the same pipeline keeps both the retrieval index and the cache current. The semantic cache sits next to agent memory in the same AI data layer: memory is durable, the cache is disposable. Context engineering explains where each store fits in what the model reads.
Pitfalls to design for
These risks follow from how similarity matching works. They are properties of the pattern rather than of one product.
- A threshold set from one dataset.
- Test it on your own prompts, since embedding models score similarity differently.
- Changing the embedding model.
- Vectors from different models are not comparable, so the cache must be emptied when the model changes.
- Caching personalised answers.
- Without scope filters, one user's answer can be served to another.
- Caching answers that depend on live data.
- Prices, stock and status change faster than most expiries.
What OSStend runs, where it runs and who it is not for
OSStend designs, moves and runs Redis and Valkey in its own cloud, in your cloud account or co-managed in your data center. Eviction and persistence are set during the design step you sign off.
Contact us if
- ✓Many users ask the same questions in different words, and model calls are a visible cost.
- ✓Cached prompts and responses are internal data that must stay in your environment.
- ✓You want the cache run alongside your other Redis or Valkey estate.
Not a fit if
- ✕Almost every prompt is unique or depends on live data, so hit rates will be low.
- ✕You cannot tolerate any chance of a near-match returning a different answer.
- ✕You want a hosted cache service and have no residency requirement. A managed service from the cache vendor may be simpler.
FAQ
What is a semantic cache?
A semantic cache stores model responses with the embedding of the prompt that produced them. A new prompt that is close enough in meaning gets the stored response, so the model is not called again.
Does Valkey support vector search for a semantic cache?
Valkey Search is a module that adds vector search to Valkey, with HNSW for approximate search and exact K-nearest-neighbour matching. It also supports numeric, tag and full-text filters, which are useful for scoping cache hits.
How do I choose a similarity threshold?
Start strict and test with real prompts, including pairs that look alike but need different answers. Loosen the threshold only while those pairs still miss. The right value depends on the embedding model, so a value from another setup does not transfer.
What is the risk of a semantic cache?
The main risk is a false hit: a stored answer returned to a question that only resembles the original. Strict thresholds, scope filters for tenant and permissions, and short expiry for changing facts reduce it.
Is a semantic cache the same as agent memory?
No. A cache is disposable and can be evicted at any time, while agent memory is durable state the agent depends on. Memory belongs in a database such as PostgreSQL, and the cache in Redis or Valkey.
Related pages
- KafkaManaged Kafka in your cloud or data centerManaged Kafka with the broker count, partition plan, KRaft, Connect and schema registry designed for your estate, run 24/7 in your cloud or your data center.
- AI data · Fresh contextEmbeddings freshness with Kafka and Postgres CDCPostgres CDC with Debezium and Kafka Connect keeps embeddings current: re-embed on change, handle deletes, and stop agents answering from stale data.
- AI data · MemoryAgent memory on PostgreSQL and Redis in productionAgent memory on PostgreSQL and Redis: long-term memory and checkpoints in Postgres with pgvector, short-term state in Redis or Valkey, and retention.
- AI dataAI database and data platform for agentsAn AI data platform built from engines you already trust: PostgreSQL with pgvector, Qdrant, OpenSearch, Redis or Valkey, Kafka and RabbitMQ, in your own environment.
- GuideContext engineering for AI agents: the data layerContext engineering for AI agents: what it is, how it differs from prompt engineering, and which data stores hold memory, retrieval and fresh context. Dated.
- Redis or ValkeyManaged Redis or Valkey, your choice of engineManaged Redis or Valkey: you choose the engine, with persistence, eviction and configuration set for your workload, run 24/7 in your cloud or data center.
Talk to an engineer about semantic caching
Bring your prompt volume and how often questions repeat. An engineer will help you judge whether a semantic cache pays off and where it should run.
- ✓A design sketch for your workload, not a plan menu
- ✓A zero-downtime cutover path for what you run today
- ✓A price for that design, after the review