AI data · Fast recall

Semantic caching with Redis or Valkey

A semantic cache that stops repeat questions from reaching the model, on Redis or Valkey that runs in OSStend's cloud, your cloud account or your data center.

45 minutes with an engineer: we review your workload and sketch the design.

A semantic cache stores model responses and returns a stored answer when a new prompt is close enough in meaning to an earlier one. It skips the model call on a hit. Redis or Valkey suit it because they combine in-memory reads, expiry and vector search.

160+AceMQ customers
10+Years running these systems
24/7Engineers on call
3Places it runs: our cloud, yours, your data center
Engines we run, on the platforms you already use
  • PostgreSQL
  • MySQL
  • MongoDB
  • Redis
  • Valkey
  • OpenSearch
  • Kafka
  • RabbitMQ
  • AWS
  • Google Cloud
  • Azure
  • Kubernetes
  • OpenShift
  • Prometheus
  • Grafana
  • Datadog
  • Dynatrace
  • Splunk

Why exact-match caching misses repeat questions

Users rarely repeat a question word for word, so an exact-match cache rarely hits. A semantic cache compares meaning instead, which raises the hit rate and introduces a new risk: returning a stored answer to a question that only looks similar.

Illustrative: plan names and shapes stand in for any provider's menu.
Paraphrases miss.
Different wording of the same question produces a different cache key.
Cost and latency repeat.
Every miss is a full model call, billed and timed again.
A loose match is a wrong answer.
Two questions can look alike and need different answers, such as the same request about two different accounts.
Stored answers go stale.
If the source data changes, a cached answer keeps repeating the old fact until it expires.
Cached answers can cross a boundary.
A response generated for one user must not be served to a user who may not see that data.
How it works

How a semantic cache works

The application embeds each incoming prompt, searches a vector index of earlier prompts for the nearest one, and compares the distance to a threshold. Under the threshold it returns the stored response. Over it, it calls the model and stores the new pair.

  1. Embed the prompt

    Turn the incoming prompt into a vector with the same embedding model used for stored prompts.

  2. Search the cache

    Run a nearest-neighbour search over cached prompt vectors, filtered by tenant or permission scope.

  3. Apply the threshold

    If the closest distance is within the threshold, return the stored response. This is a hit.

  4. Call the model on a miss

    Otherwise call the model as usual, return the answer and store the prompt vector with the response and an expiry.

ComponentRoleDocumentation
Redis or Valkey key-value storeHolds responses, with expiry and a memory limitRedis LangCache describes configurable similarity thresholds, TTLs and eviction policies
Vector indexFinds the nearest cached promptValkey Search is a module with HNSW approximate search and exact KNN
Filter fieldsScope hits to a tenant, model or languageValkey Search supports tag, numeric and full-text filters alongside vector queries
Embedding modelProduces prompt vectorsRuns outside the cache; the model must be the same for stored and incoming prompts

Sources: Redis and Valkey documentation, checked 30 Oct 2026. Redis describes LangCache as in preview on that date, so check current status.

Thresholds, eviction and cost

Three settings decide whether a semantic cache pays off: the similarity threshold, the expiry and eviction policy, and the hit rate they produce. The cost maths is a comparison of saved model calls against the cost of embedding every prompt and running the cache.

Setting or driverWhat it changesHow to approach it
Similarity thresholdHit rate against wrong-answer riskStart strict. Test with real prompt pairs, including near-misses that need different answers, and loosen only while those still miss.
Expiry (TTL)StalenessSet by how fast the underlying facts change. Invalidate on source changes where possible.
Eviction policy and memory limitWhich entries survive when memory is fullSet a memory limit and an eviction policy that matches your access pattern. Cache entries can always be rebuilt.
PersistenceCold-start behaviourOptional for a cache. Persisting avoids an empty cache after a restart, at some write cost.
Scope filtersSafety across users and tenantsInclude tenant, permission group and model version in the lookup.
Cost mathsNet savingSaving = hit rate × requests × cost of a model call, minus the embedding cost for every request and the cost of running the cache. Measure hit rate before committing.

Cache settings follow the engine documentation: Redis LangCache and Valkey Search, checked 30 Oct 2026.

Keeping cached answers fresh

A cached answer is only as current as the data behind it. Use a short expiry for fast-changing facts, and where the source is a database, publish its changes so affected entries can be removed.

A change stream from the source database into Kafka can drive invalidation as well as re-embedding (see keeping embeddings fresh with Kafka and CDC), so the same pipeline keeps both the retrieval index and the cache current. The semantic cache sits next to agent memory in the same AI data layer: memory is durable, the cache is disposable. Context engineering explains where each store fits in what the model reads.

Pitfalls to design for

These risks follow from how similarity matching works. They are properties of the pattern rather than of one product.

A threshold set from one dataset.
Test it on your own prompts, since embedding models score similarity differently.
Changing the embedding model.
Vectors from different models are not comparable, so the cache must be emptied when the model changes.
Caching personalised answers.
Without scope filters, one user's answer can be served to another.
Caching answers that depend on live data.
Prices, stock and status change faster than most expiries.
Enterprise only

What OSStend runs, where it runs and who it is not for

OSStend designs, moves and runs Redis and Valkey in its own cloud, in your cloud account or co-managed in your data center. Eviction and persistence are set during the design step you sign off.

Contact us if

  • Many users ask the same questions in different words, and model calls are a visible cost.
  • Cached prompts and responses are internal data that must stay in your environment.
  • You want the cache run alongside your other Redis or Valkey estate.
Talk to an engineer

Not a fit if

  • Almost every prompt is unique or depends on live data, so hit rates will be low.
  • You cannot tolerate any chance of a near-match returning a different answer.
  • You want a hosted cache service and have no residency requirement. A managed service from the cache vendor may be simpler.

What is a semantic cache?

A semantic cache stores model responses with the embedding of the prompt that produced them. A new prompt that is close enough in meaning gets the stored response, so the model is not called again.

Does Valkey support vector search for a semantic cache?

Valkey Search is a module that adds vector search to Valkey, with HNSW for approximate search and exact K-nearest-neighbour matching. It also supports numeric, tag and full-text filters, which are useful for scoping cache hits.

How do I choose a similarity threshold?

Start strict and test with real prompts, including pairs that look alike but need different answers. Loosen the threshold only while those pairs still miss. The right value depends on the embedding model, so a value from another setup does not transfer.

What is the risk of a semantic cache?

The main risk is a false hit: a stored answer returned to a question that only resembles the original. Strict thresholds, scope filters for tenant and permissions, and short expiry for changing facts reduce it.

Is a semantic cache the same as agent memory?

No. A cache is disposable and can be evicted at any time, while agent memory is durable state the agent depends on. Memory belongs in a database such as PostgreSQL, and the cache in Redis or Valkey.

Talk to an engineer

Talk to an engineer about semantic caching

Bring your prompt volume and how often questions repeat. An engineer will help you judge whether a semantic cache pays off and where it should run.

  • A design sketch for your workload, not a plan menu
  • A zero-downtime cutover path for what you run today
  • A price for that design, after the review
See pricing →

Your request goes to the OSStend team. No newsletter sign-up.

Get a quoteTalk to an engineer
2Engines
3Where should it run?
4What do you need?

Your request goes to the OSStend team. No newsletter sign-up.