Keeping embeddings fresh with Kafka and change data capture
Re-embed on change, so an agent never answers from stale context. The pipeline runs on Kafka and PostgreSQL in your cloud, your data center or ours.
45 minutes with an engineer: we review your workload and sketch the design.
Postgres CDC, or change data capture, reads each insert, update and delete from the database log and publishes it to Kafka. A consumer then re-embeds only the rows that changed and updates the vector index. Agents and retrieval see current data without a nightly full re-index.
- PostgreSQL
- MySQL
- MongoDB
- Redis
- Valkey
- OpenSearch
- Kafka
- RabbitMQ
- AWS
- Google Cloud
- Azure
- Kubernetes
- OpenShift
- Prometheus
- Grafana
- Datadog
- Dynatrace
- Splunk
Why embeddings go stale
An embedding is a snapshot of text at one moment. When the source row changes and the vector does not, retrieval returns the old meaning and the agent answers confidently from it.
- Batch re-indexing lags.
- A nightly job leaves a window where answers are out of date, and a full re-index re-embeds rows that did not change.
- Dual writes drift.
- If the application writes the row and the vector separately, one of the two writes can fail and the stores disagree.
- Deleted rows linger.
- A vector for a deleted or restricted row can keep surfacing in results.
- Model changes strand old vectors.
- Vectors from different embedding models are not comparable, so a model change means re-embedding everything.
- Cost scales with rows, not changes.
- Re-embedding the whole table every time is the most expensive way to stay current.
The pattern: log-based change data capture into Kafka
The database log is the source of truth for what changed. A connector reads it, publishes one event per change to Kafka, and a consumer turns events into vector updates.
Capture
The Debezium PostgreSQL connector reads committed changes through PostgreSQL logical decoding, with the native `pgoutput` plug-in available from PostgreSQL 10.
Publish
By default each table gets its own Kafka topic. Each event carries the row key, the `before` and `after` states, and an `op` field for create, update, delete or snapshot read.
Embed
A consumer reads the topic, skips events where the embedded text did not change, and calls the embedding model for the rest.
Upsert or delete
The vector is written to the index under the row's primary key. A delete event removes it.
| Component | Role | Documentation |
|---|---|---|
| PostgreSQL logical decoding | Streams committed changes from the write-ahead log | Logical decoding |
| Debezium PostgreSQL connector | Captures changes and emits change events, with a snapshot of existing rows on first start | Debezium PostgreSQL connector |
| Kafka and Kafka Connect | Durable, ordered-per-partition event log, and the framework that runs connectors | Kafka Connect |
| Embedding consumer | Reads events, calls the embedding model, writes vectors | Your code or a stream-processing job; runs outside Kafka |
| Vector index | Stores vectors for retrieval, for example in PostgreSQL with pgvector | pgvector; see also pgvector on OSStend |
Documentation checked 30 Oct 2026.
Settings that decide whether it holds up
Most of the tuning sits in the connector and the consumer. A few settings are easy to miss and cause the problems seen in production, such as write-ahead log growth.
| Setting or driver | What it changes | How to approach it |
|---|---|---|
| Replication slot | Where the connector resumes, and how much write-ahead log the server keeps | Debezium recommends creating the slot manually in production. Monitor slot lag, since an unread slot makes WAL accumulate. |
| Heartbeats | Whether the connector acknowledges progress on quiet databases | Debezium documents heartbeats and a heartbeat action query for low-traffic databases, to stop WAL files accumulating. |
| Snapshot mode | How existing rows are loaded first | The default takes a consistent snapshot, then streams changes. Plan the initial embedding run, which is the largest cost. |
| `REPLICA IDENTITY` | What the `before` state contains | Debezium notes `FULL` is needed in some cases, such as keys that are not the primary key. Decide per table. |
| TOAST columns | Large values that are unchanged may not be sent | Debezium substitutes a placeholder for an unchanged toasted value. The consumer must not treat it as new text. |
| Topic partitions and keys | Ordering and parallelism | Events with the same key stay in order within a partition. Size partitions for the consumer count. |
| Embedding batch size and rate | Latency, cost and model rate limits | Batch calls, cap the rate and make writes idempotent so retries are safe. |
| Consumer lag | How stale the index is | Track lag as the freshness measure, and alert when it grows. |
Connector behaviour from the Debezium documentation, checked 30 Oct 2026.
The cost of staying fresh is the embedding calls for changed rows, plus the cluster that carries the events. Skipping events where the embedded text did not change cuts the first part. Comparing the `before` and `after` states needs `REPLICA IDENTITY FULL` on the table; otherwise compare a stored hash of the text.
Pitfalls to design for
These follow from how change data capture and embeddings work. They are properties of the pattern rather than of one product.
- An idle replication slot.
- If the connector stops, the database keeps the log it has not delivered, and disk can fill.
- Treating a placeholder as text.
- A value marked unavailable must keep the existing vector, not be embedded.
- Ignoring deletes.
- Without handling the delete event, removed or restricted rows stay searchable.
- Schema changes.
- A new column or a changed type needs a plan for the consumer and the index.
- Changing the embedding model mid-stream.
- Old and new vectors are not comparable, so plan a full re-embed with a cut-over.
- At-least-once delivery.
- The same event can arrive twice, so upserts by primary key must be safe to repeat.
What OSStend runs, where it runs and who it is not for
OSStend designs, moves and runs Kafka and PostgreSQL in its own cloud, in your cloud account or co-managed in your data center. Broker count, partition plan and the change data capture setup are part of the cluster design you sign off.
The pipeline feeds the retrieval side of the AI data layer, such as vector search for RAG. OSStend runs the Debezium side as managed Debezium; for the tools and methods, read change data capture explained.
Contact us if
- ✓Source data changes through the day and answers must reflect it.
- ✓The source data and the vectors have to stay in your environment.
- ✓You already run, or plan to run, Kafka for other events.
Not a fit if
- ✕The data changes rarely. A scheduled re-index is simpler.
- ✕You have no database you can enable logical decoding on.
- ✕The corpus is documents in files rather than rows in a database. A file-watching or batch pipeline fits better.
FAQ
What is Postgres CDC?
Postgres CDC, or change data capture, reads committed changes from the PostgreSQL write-ahead log through logical decoding and publishes them as events. Debezium is the common open-source connector for it.
How does Debezium work with Kafka?
Debezium runs as a Kafka Connect connector. It reads database changes and writes one event per change to a Kafka topic, by default one topic per table.
How do I keep embeddings up to date?
Capture changes, publish them to Kafka, and have a consumer re-embed only rows whose embedded text changed. Write the vector under the row's primary key, and remove it on a delete event.
What happens to embeddings when a row is deleted?
The delete event from change data capture tells the consumer to remove the vector. Without that step the vector remains searchable.
Does change data capture put load on the database?
Logical decoding reads the write-ahead log rather than querying tables. The main operational risk is write-ahead log growth if the connector stops, so slot lag needs monitoring.
Related pages
- AI data · PostgreSQLpgvector in production: indexes and sizingpgvector in production: HNSW against IVFFlat, memory sizing, recall against latency, filtering, replicas and pgvectorscale, with links to the pgvector docs.
- KafkaManaged Kafka in your cloud or data centerManaged Kafka with the broker count, partition plan, KRaft, Connect and schema registry designed for your estate, run 24/7 in your cloud or your data center.
- PostgreSQLManaged PostgreSQL service and Postgres hostingManaged PostgreSQL service, compared: how to choose between RDS, Aiven, Crunchy Bridge, Neon and a tailored cluster run 24/7 in your cloud or ours.
- AI dataAI database and data platform for agentsAn AI data platform built from engines you already trust: PostgreSQL with pgvector, Qdrant, OpenSearch, Redis or Valkey, Kafka and RabbitMQ, in your own environment.
- AI data · RetrievalVector database for RAG on your own infrastructureA vector database for RAG on your own infrastructure: when PostgreSQL with pgvector is enough, when OpenSearch hybrid search fits, and how to size retrieval.
- Change data captureManaged Debezium: change data capture run 24/7Managed change data capture with Debezium on Kafka Connect or Debezium Server, run 24/7 in OSStend's cloud, your cloud account or your data center.
Talk to an engineer about fresh embeddings
Bring the tables your agents read from and how often they change. An engineer will help you judge whether change data capture fits and where it should run.
- ✓A design sketch for your workload, not a plan menu
- ✓A zero-downtime cutover path for what you run today
- ✓A price for that design, after the review