AI data · Fresh context

Keeping embeddings fresh with Kafka and change data capture

Re-embed on change, so an agent never answers from stale context. The pipeline runs on Kafka and PostgreSQL in your cloud, your data center or ours.

45 minutes with an engineer: we review your workload and sketch the design.

Postgres CDC, or change data capture, reads each insert, update and delete from the database log and publishes it to Kafka. A consumer then re-embeds only the rows that changed and updates the vector index. Agents and retrieval see current data without a nightly full re-index.

160+AceMQ customers
10+Years running these systems
24/7Engineers on call
3Places it runs: our cloud, yours, your data center
Engines we run, on the platforms you already use
  • PostgreSQL
  • MySQL
  • MongoDB
  • Redis
  • Valkey
  • OpenSearch
  • Kafka
  • RabbitMQ
  • AWS
  • Google Cloud
  • Azure
  • Kubernetes
  • OpenShift
  • Prometheus
  • Grafana
  • Datadog
  • Dynatrace
  • Splunk

Why embeddings go stale

An embedding is a snapshot of text at one moment. When the source row changes and the vector does not, retrieval returns the old meaning and the agent answers confidently from it.

Illustrative: similar documents sit close together, so retrieval is a nearest-neighbour search.
Batch re-indexing lags.
A nightly job leaves a window where answers are out of date, and a full re-index re-embeds rows that did not change.
Dual writes drift.
If the application writes the row and the vector separately, one of the two writes can fail and the stores disagree.
Deleted rows linger.
A vector for a deleted or restricted row can keep surfacing in results.
Model changes strand old vectors.
Vectors from different embedding models are not comparable, so a model change means re-embedding everything.
Cost scales with rows, not changes.
Re-embedding the whole table every time is the most expensive way to stay current.
How it works

The pattern: log-based change data capture into Kafka

The database log is the source of truth for what changed. A connector reads it, publishes one event per change to Kafka, and a consumer turns events into vector updates.

  1. Capture

    The Debezium PostgreSQL connector reads committed changes through PostgreSQL logical decoding, with the native `pgoutput` plug-in available from PostgreSQL 10.

  2. Publish

    By default each table gets its own Kafka topic. Each event carries the row key, the `before` and `after` states, and an `op` field for create, update, delete or snapshot read.

  3. Embed

    A consumer reads the topic, skips events where the embedded text did not change, and calls the embedding model for the rest.

  4. Upsert or delete

    The vector is written to the index under the row's primary key. A delete event removes it.

ComponentRoleDocumentation
PostgreSQL logical decodingStreams committed changes from the write-ahead logLogical decoding
Debezium PostgreSQL connectorCaptures changes and emits change events, with a snapshot of existing rows on first startDebezium PostgreSQL connector
Kafka and Kafka ConnectDurable, ordered-per-partition event log, and the framework that runs connectorsKafka Connect
Embedding consumerReads events, calls the embedding model, writes vectorsYour code or a stream-processing job; runs outside Kafka
Vector indexStores vectors for retrieval, for example in PostgreSQL with pgvectorpgvector; see also pgvector on OSStend

Documentation checked 30 Oct 2026.

Settings that decide whether it holds up

Most of the tuning sits in the connector and the consumer. A few settings are easy to miss and cause the problems seen in production, such as write-ahead log growth.

Illustrative shape: real designs start from your metrics.
Setting or driverWhat it changesHow to approach it
Replication slotWhere the connector resumes, and how much write-ahead log the server keepsDebezium recommends creating the slot manually in production. Monitor slot lag, since an unread slot makes WAL accumulate.
HeartbeatsWhether the connector acknowledges progress on quiet databasesDebezium documents heartbeats and a heartbeat action query for low-traffic databases, to stop WAL files accumulating.
Snapshot modeHow existing rows are loaded firstThe default takes a consistent snapshot, then streams changes. Plan the initial embedding run, which is the largest cost.
`REPLICA IDENTITY`What the `before` state containsDebezium notes `FULL` is needed in some cases, such as keys that are not the primary key. Decide per table.
TOAST columnsLarge values that are unchanged may not be sentDebezium substitutes a placeholder for an unchanged toasted value. The consumer must not treat it as new text.
Topic partitions and keysOrdering and parallelismEvents with the same key stay in order within a partition. Size partitions for the consumer count.
Embedding batch size and rateLatency, cost and model rate limitsBatch calls, cap the rate and make writes idempotent so retries are safe.
Consumer lagHow stale the index isTrack lag as the freshness measure, and alert when it grows.

Connector behaviour from the Debezium documentation, checked 30 Oct 2026.

The cost of staying fresh is the embedding calls for changed rows, plus the cluster that carries the events. Skipping events where the embedded text did not change cuts the first part. Comparing the `before` and `after` states needs `REPLICA IDENTITY FULL` on the table; otherwise compare a stored hash of the text.

Pitfalls to design for

These follow from how change data capture and embeddings work. They are properties of the pattern rather than of one product.

An idle replication slot.
If the connector stops, the database keeps the log it has not delivered, and disk can fill.
Treating a placeholder as text.
A value marked unavailable must keep the existing vector, not be embedded.
Ignoring deletes.
Without handling the delete event, removed or restricted rows stay searchable.
Schema changes.
A new column or a changed type needs a plan for the consumer and the index.
Changing the embedding model mid-stream.
Old and new vectors are not comparable, so plan a full re-embed with a cut-over.
At-least-once delivery.
The same event can arrive twice, so upserts by primary key must be safe to repeat.
Enterprise only

What OSStend runs, where it runs and who it is not for

OSStend designs, moves and runs Kafka and PostgreSQL in its own cloud, in your cloud account or co-managed in your data center. Broker count, partition plan and the change data capture setup are part of the cluster design you sign off.

The pipeline feeds the retrieval side of the AI data layer, such as vector search for RAG. OSStend runs the Debezium side as managed Debezium; for the tools and methods, read change data capture explained.

Contact us if

  • Source data changes through the day and answers must reflect it.
  • The source data and the vectors have to stay in your environment.
  • You already run, or plan to run, Kafka for other events.
Talk to an engineer

Not a fit if

  • The data changes rarely. A scheduled re-index is simpler.
  • You have no database you can enable logical decoding on.
  • The corpus is documents in files rather than rows in a database. A file-watching or batch pipeline fits better.

What is Postgres CDC?

Postgres CDC, or change data capture, reads committed changes from the PostgreSQL write-ahead log through logical decoding and publishes them as events. Debezium is the common open-source connector for it.

How does Debezium work with Kafka?

Debezium runs as a Kafka Connect connector. It reads database changes and writes one event per change to a Kafka topic, by default one topic per table.

How do I keep embeddings up to date?

Capture changes, publish them to Kafka, and have a consumer re-embed only rows whose embedded text changed. Write the vector under the row's primary key, and remove it on a delete event.

What happens to embeddings when a row is deleted?

The delete event from change data capture tells the consumer to remove the vector. Without that step the vector remains searchable.

Does change data capture put load on the database?

Logical decoding reads the write-ahead log rather than querying tables. The main operational risk is write-ahead log growth if the connector stops, so slot lag needs monitoring.

Talk to an engineer

Talk to an engineer about fresh embeddings

Bring the tables your agents read from and how often they change. An engineer will help you judge whether change data capture fits and where it should run.

  • A design sketch for your workload, not a plan menu
  • A zero-downtime cutover path for what you run today
  • A price for that design, after the review
See pricing →

Your request goes to the OSStend team. No newsletter sign-up.

Get a quoteTalk to an engineer
2Engines
3Where should it run?
4What do you need?

Your request goes to the OSStend team. No newsletter sign-up.