Skip to main content

> change_data_capture_(cdc)_with_debezium_&_kafka

Change Data Capture (CDC) with Debezium & Kafka

How does log-based Change Data Capture (CDC) extract commit-log mutations directly from database engines without degrading application throughput or locking OLTP tables?

Stack: SOFTWARE ARCHITECTURE STACK →Staff (L6-L7)pattern

THE SHORT ANSWER

By hooking into low-level database replication logs (PostgreSQL WAL via logical decoding or MySQL row-based binlog) as a pseudo-replica and streaming parsed row-level mutations directly into Apache Kafka topics via Kafka Connect.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

Debezium runs as a source connector inside the Kafka Connect runtime. It connects to the primary database using native replication protocols. Rather than executing periodic SQL SELECT queries (which cause table scans and miss intermediary deletes/updates), Debezium parses the raw binary transaction log. Every committed INSERT, UPDATE, and DELETE is converted into a strongly typed event payload containing both 'before' and 'after' row states, transaction metadata, and source sequence offsets, and published to a partitioned Kafka topic.

2. Appropriate Use Context

Real-time search index synchronization (Elasticsearch/OpenSearch), cache invalidation pipelines (Redis), data lake ingestion (Snowflake/Iceberg), outbox event relaying, and zero-impact event-driven microservice integration.

3. Production Failure Modes

1) WAL Disk Overflow: A lagging or disconnected Debezium connector prevents the database from purging WAL segments, filling the database disk and crashing production; 2) Schema Evolution Mismatch: Dropping or altering columns causing Debezium to crash on unexpected schema tokens; 3) Initial Snapshot Saturation: Large initial table snapshots exhausting database connection pools and memory.

4. Diagnostic Signals & Telemetry

Monitoring PostgreSQL replication slot lag (bytes behind master), Kafka Connect connector status (FAILED vs RUNNING), Debezium event processing delay (msec between DB commit and Kafka record timestamp), and database disk free space.

5. Prevention & Safeguards

Configure `max_slot_wal_keep_size` in PostgreSQL to prevent runaway disk exhaustion, integrate Confluent Schema Registry with Avro/Protobuf for safe schema evolution, and run incremental read-only chunked snapshots for large legacy tables.

6. Architectural Trade-offs

Provides sub-second, zero-application-overhead data streaming at the expense of managing Kafka Connect infrastructure and monitoring sensitive database replication slot limits.

Case Study (TinyCTO In-Field Example)

TinyCTO Architecture: A product search engine was previously refreshed via a nightly SQL dump script that ran for 4 hours and degraded checkout performance. Deploying Debezium CDC on the catalog database enabled real-time search index updates within 120ms of a merchant price change, reducing origin database CPU load by 74%.

Interactive Concept Drills

3 Cards
Q1

How does log-based CDC differ from query-based polling CDC (e.g. SELECT WHERE updated_at > t)?

Log-based CDC captures every intermediate update, transaction metadata, and hard DELETEs directly from transaction logs without running CPU-heavy table scans on the primary database.
Q2

What catastrophic incident occurs if a PostgreSQL replication slot used by Debezium is abandoned?

PostgreSQL stops pruning WAL log files, retaining all transaction logs indefinitely until the host storage volume is 100% full, causing a total database crash.
Q3

What information is typically contained in a Debezium CDC event payload?

The 'before' state of the row, the 'after' state, operation type (c=create, u=update, d=delete), transaction commit timestamp, and database log sequence number (LSN/offset).

Change Data Capture (CDC) with Debezium & Kafka — Technical FAQ

Can Debezium stream data from replica nodes instead of the primary master?

In PostgreSQL, logical decoding requires writing to replication slots which is traditionally only supported on primary nodes (though PG16+ introduces logical decoding on standbys). MySQL supports binlog CDC from replicas.

How does Debezium handle database schema migrations (DDL changes)?

Debezium captures DDL statements, updates its internal schema history topic, and emits updated Avro/JSON schemas to the Schema Registry so consumers adapt seamlessly.

Is Debezium CDC event ordering guaranteed in Kafka?

Yes, provided that events are partitioned using the table's Primary Key. Kafka guarantees strict FIFO ordering within a single partition.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸Debezium is built on the Apache Kafka Connect framework and provides production-grade connectors for PostgreSQL, MySQL, SQL Server, Oracle, and MongoDB.
  • ▸Log-based CDC provides a true immutable change stream representing 100% of state transitions, unlike periodic snapshot polling.

Common Misconceptions

  • ✗Assuming CDC is only for big data analytics; it is widely used for core operational workflows like search synchronization, cache invalidation, and transactional outbox relaying.

Decision & Governance Guidance

Deploy Debezium CDC when downstream consumers need real-time data replication without placing query load or code changes on the primary operational database.

Authoritative Sources & Standards