Change data capture: what it is, which tools to use and what runs it
Change data capture turns database changes into a stream of events. This guide covers how it works, the CDC tools teams compare, what Postgres CDC needs, and when running it yourself stops being worth it.
- ✓Checked 18 Nov 2026
- ✓OSStend runs Debezium
Change data capture (CDC) records each insert, update and delete in a source database and delivers it to other systems. Log-based CDC reads the database's transaction log and is the usual choice for production. Debezium, AWS DMS and several managed platforms offer it. Descriptions checked 18 Nov 2026 from their own pages; nothing was benchmarked.
What is change data capture?
Change data capture is a way to find out what changed in a database and send those changes onward as they happen, instead of copying whole tables on a schedule. The destination can be a warehouse, a search index, a cache, a message broker or another database.
A CDC pipeline has three parts: a capture step that reads changes at the source, a transport that carries them (often a broker such as Kafka), and a consumer that applies them. Common uses are keeping a warehouse or search index current, feeding other services, replicating between databases and keeping AI embeddings fresh.
Log-based vs query-based vs trigger-based CDC
There are three ways to capture changes. Log-based CDC reads the database's transaction log and is the one that sees every change, deletes included, with the least load on the source. Query-based and trigger-based capture are simpler to start and have clear limits.
| Method | How it works | Strengths | Limits |
|---|---|---|---|
| Log-based | Reads the transaction log (WAL, binlog, redo log or the database's CDC tables). | Sees inserts, updates and deletes in commit order; no change to application tables; low load on the source. | Needs database settings and privileges, and the log has to be kept until the reader has read it. |
| Query-based | Polls tables on a schedule, usually by an updated-at column or an id. | No special database setup; easy to reason about. | Misses hard deletes and changes between polls; every poll queries the source; needs a reliable column. |
| Trigger-based | Triggers write each change to a side table that a process reads. | Works where logs are not readable. | Adds work to every write, and triggers must be maintained with the schema. |
General description of the three methods. A tool may combine them; check which one a given connector uses.
CDC tools: how the main options differ
CDC tools fall into open-source engines you operate, cloud migration services and managed data-movement platforms. They differ in how they capture, where changes can go and who runs them, which matters more than the feature list.
| Tool | What it is | Where changes go | Who operates it |
|---|---|---|---|
| Debezium | Open-source, log-based CDC platform with connectors for PostgreSQL, MySQL, SQL Server, Oracle, MongoDB, Db2 and others (connectors, checked 18 Nov 2026). | Kafka through Kafka Connect, or other systems through Debezium Server (sinks). | You, or a provider such as OSStend |
| AWS DMS | An AWS service that migrates data and replicates ongoing changes between supported sources and targets (documentation, checked 18 Nov 2026). | A fixed list of targets that includes databases, Amazon S3, Amazon Kinesis and Apache Kafka (targets). | AWS runs the replication instance; you configure tasks |
| Fivetran and Airbyte | Data-integration platforms that load data into warehouses and lakes, with change capture options for some database sources. | Warehouses, lakes and other destinations the platform supports. | The vendor for hosted plans; you for self-hosted Airbyte |
| Estuary and Streamkap | Managed platforms for real-time data movement and CDC streaming. | The destinations each platform supports. | The vendor |
| Confluent Cloud connectors | Fully managed source connectors, including database CDC connectors, inside Confluent Cloud. | Kafka topics on Confluent Cloud. | Confluent |
Debezium and AWS DMS are described from their documentation, read 18 Nov 2026. The platform rows are category descriptions, not a feature comparison: each vendor's own connector list is current and ours is not. We list no prices because they change and differ by unit. OSStend runs Debezium, so weigh this as a vendor's view.
For a detailed look at the two tools teams compare most, read Debezium vs AWS DMS.
Postgres CDC: logical replication, replication slots and WAL
Postgres CDC reads committed changes through logical decoding, which turns the write-ahead log (WAL) into row-level changes. A replication slot remembers how far a consumer has read, and PostgreSQL keeps WAL until the slot's consumer confirms it, so a stalled consumer can fill the disk.
- wal_level must be logical
- for logical decoding, which needs a server restart if it was not already set (PostgreSQL docs, checked 18 Nov 2026).
- A slot holds WAL.
- The server keeps the WAL a slot still needs, so a connector that is down or slow makes WAL accumulate on the primary.
- A cap exists.
- The max_slot_wal_keep_size setting limits how much WAL a slot may retain; past it the slot is invalidated and the consumer must re-snapshot (replication settings, checked 18 Nov 2026).
- Debezium creates and reads the slot.
- Its PostgreSQL connector documents using a slot with the pgoutput plug-in and recommends watching slot lag; heartbeats stop WAL building up on quiet databases (connector docs, checked 18 Nov 2026).
- Failover matters.
- After a primary failover, a slot that was not carried to the new primary has to be recreated, and the changes in between need a re-snapshot or a replay plan.
The managed PostgreSQL page covers how OSStend designs replicas and failover, and the embeddings freshness guide walks through Postgres CDC with Debezium and Kafka.
From our on-call engineers: what breaks first in CDC
CDC is easy to demo and hard to keep right. In the pipelines we have designed and reviewed, the trouble was almost never the connector's feature set: it was log retention, recovery and upgrades.
- ✓Log retention has to cover the longest outage you plan for. While the capture tool is down, the source must keep its change logs until it catches up.
- ✓Recovery from the last checkpoint should be tested, with the connector killed on purpose and a broker outage simulated, not only read about.
- ✓Deletes are the event type most often missed in a first proof of concept, so test them first.
- ✓Disaster recovery that depends on copying offsets by hand will fail on the day it is needed; offsets belong in the recovery design.
- ✓Upgrades of the capture tool and the broker beneath it are one change, and teams that postpone them for fear of breaking the pipeline pile up risk.
- ✓Alternative paths have their own limits: a migration service that reads transaction logs can lose access when the source's maintenance removes those logs.
When should you pay someone to run CDC?
Pay someone to run CDC when the pipeline matters to the business and nobody on your team owns it at 3 a.m. Running it yourself is reasonable while it feeds a dashboard that can be a day stale.
| Your situation | Usual fit |
|---|---|
| A pipeline feeds a report that can be refreshed overnight, and one engineer knows it | Run it yourself |
| Lag has reached hours, events are missing or recovery has never been tested | An engineer-led review, then managed operations |
| The platform team will not deploy a CDC tool without a support arrangement | A managed operations contract. See managed Debezium |
| You only need a one-time database migration | A migration service or a dump and restore, not a CDC platform |
| Many consumers read the same changes, and Kafka already runs | Debezium on Kafka Connect, run with managed Kafka |
FAQ
What is change data capture in simple terms?
It is reading each insert, update and delete in a database as it happens and sending it on, instead of copying whole tables on a schedule.
What are the best CDC tools?
It depends on where the changes must go and who runs it. Debezium is the common open-source choice, AWS DMS fits AWS-centred replication, and managed platforms such as Fivetran, Airbyte, Estuary, Streamkap and Confluent Cloud connectors fit teams that do not want to operate it. See the tools table.
How does Postgres CDC work?
A tool reads PostgreSQL's WAL through logical decoding, using a replication slot to remember its position. The slot keeps WAL on the primary until the tool confirms it, so slot lag has to be monitored.
Is CDC the same as ETL?
No. ETL copies and transforms data in batches. CDC streams individual changes. CDC often feeds an ETL or ELT load, and the two are used together.
Is CDC the same as database replication?
They overlap. Physical replication copies the whole database for a standby. CDC delivers row-level changes in a form other systems can consume, so the destination need not be the same database.
Does CDC need Kafka?
No. Kafka is a common transport, and Debezium Server can write to RabbitMQ, Amazon Kinesis and Google Cloud Pub/Sub without it. Managed Debezium covers both.
Debezium is a trademark of its respective owner and Apache Kafka is a trademark of the Apache Software Foundation. OSStend is not affiliated with or endorsed by these projects. Other product names belong to their owners.
Related pages
- AI data · Fresh contextEmbeddings freshness with Kafka and Postgres CDCPostgres CDC with Debezium and Kafka Connect keeps embeddings current: re-embed on change, handle deletes, and stop agents answering from stale data.
- Change data captureManaged Debezium: change data capture run 24/7Managed change data capture with Debezium on Kafka Connect or Debezium Server, run 24/7 in OSStend's cloud, your cloud account or your data center.
- ComparisonDebezium vs AWS DMS: change data capture comparedDebezium vs AWS DMS for change data capture: sources, targets, operations, log handling and when each fits. Read from both vendors' documentation, 18 Nov 2026.
- PostgreSQLManaged PostgreSQL service and Postgres hostingManaged PostgreSQL service, compared: how to choose between RDS, Aiven, Crunchy Bridge, Neon and a tailored cluster run 24/7 in your cloud or ours.
- KafkaManaged Kafka in your cloud or data centerManaged Kafka with the broker count, partition plan, KRaft, Connect and schema registry designed for your estate, run 24/7 in your cloud or your data center.
Take your CDC pipeline to an engineer
45 minutes: your sources, change volume and recovery needs are reviewed, and an engineer says what to run and what to hand over.
- ✓A design sketch for your workload, not a plan menu
- ✓A zero-downtime cutover path for what you run today
- ✓A price for that design, after the review