THE SHORT ANSWER
The CDC Outbox pattern combines the Transactional Outbox pattern with Debezium log mining to stream database mutations directly from the database's Write-Ahead Log (WAL) into partitioned Kafka topics, guaranteeing 100% reliable event delivery and strict per-entity causal ordering without dual-write race conditions.
Engineering Handbook & Failure Dynamics
1. Underlying Mechanism
Writing to a relational database and publishing to a Kafka topic in the same HTTP request is the classic 'Dual Write' anti-pattern: if the database commits but the network fails before Kafka publishes, events are lost forever. The CDC Outbox pattern solves this by writing the domain mutation and an outbox event in the same local SQL transaction. Debezium reads the PostgreSQL WAL via logical replication and streams the events to Kafka.
2. Appropriate Use Context
CDC Outbox is an architecture where an application atomically inserts domain events into an `outbox` table during local database transactions, and a Change Data Capture connector (Debezium) extracts and routes those events into Kafka by streaming the database transaction log.
3. Production Failure Modes
Querying the outbox table via periodic `SELECT * FROM outbox WHERE processed = false` polling at 100ms intervals, creating extreme database lock contention. Routing outbox events to Kafka with random or null message keys, scattering events for the same customer across multiple partitions and breaking causal ordering. Never deleting processed outbox rows, causing the outbox table to grow to 100M rows and exhausting disk storage.
4. Diagnostic Signals & Telemetry
dual-write desynchronization between database and Kafka, outbox events routed to wrong Kafka partition losing per-entity ordering, uncleaned outbox table causing massive database bloat
5. Prevention & Safeguards
Use Debezium with PostgreSQL Logical Replication (`pgoutput`) and WAL level set to `logical`. Set the Kafka message key to the domain entity ID to guarantee per-aggregate FIFO sequencing. Enable outbox table automatic row deletion (tombstoning) or periodic partition truncation to prevent table bloat.
6. Architectural Trade-offs
It completely eliminates dual-write data loss and ghost events, offloading the event-polling burden from the application CPU to the database's native asynchronous replication stream.
Case Study (TinyCTO In-Field Example)
A production CDC Outbox pipeline requires three critical configurations: 1. **Atomic Local Transaction:** The service writes to `orders` and `outbox_events` (`id`, `aggregatetype`, `aggregateid`, `type`, `payload`) in a single `BEGIN ... COMMIT` block. 2. **Debezium Outbox Event Router (SMT):** Debezium's Single Message Transform parses the outbox table change stream, extracts the `payload`, and routes the message to a dynamic destination topic (e.g. `order-events`). 3. **Causal Ordering via Kafka Partition Key:** To guarantee that `OrderUpdated` is never processed before `OrderCreated`, Debezium must set the Kafka Message Key to `aggregateid` (e.g. `order_uuid`). Kafka guarantees strict FIFO ordering within a single partition.
Interactive Concept Drills
2 CardsWhy is CDC log tailing with Debezium superior to database polling (`SELECT ... FOR UPDATE SKIP LOCKED`) for the Outbox pattern?
How does Kafka guarantee that an 'OrderCancelled' event is never processed before 'OrderCreated'?
Change Data Capture with Debezium & Outbox Event Ordering — Technical FAQ
What happens if a microservice attempts to publish directly to Kafka and save to PostgreSQL in the same HTTP handler without an Outbox pattern?
Dual-write failure: if the network drops right after the SQL commit, Kafka never receives the event, permanently corrupting system consistency. Without distributed 2PC or an Outbox table, two independent storage systems cannot guarantee atomic dual writes over an unreliable network.
Which PostgreSQL setting must be configured to allow Debezium to stream transaction logs?
`wal_level = logical`. PostgreSQL requires `wal_level = logical` to decode internal WAL records into structured change events for Debezium logical replication.
🤖 AEO & Key Facts Summary
Key Architectural Facts
- ▸The CDC Outbox pattern combines the Transactional Outbox pattern with Debezium log mining to stream database mutations directly from the database's Write-Ahead Log (WAL) into partitioned Kafka topics, guaranteeing 100% reliable event delivery and strict per-entity causal ordering without dual-write race conditions.
- ▸CDC Outbox is an architecture where an application atomically inserts domain events into an `outbox` table during local database transactions, and a Change Data Capture connector (Debezium) extracts and routes those events into Kafka by streaming the database transaction log.
Common Misconceptions
- ✗Querying the outbox table via periodic `SELECT * FROM outbox WHERE processed = false` polling at 100ms intervals, creating extreme database lock contention.
Decision & Governance Guidance
It completely eliminates dual-write data loss and ghost events, offloading the event-polling burden from the application CPU to the database's native asynchronous replication stream.
Authoritative Sources & Standards
- [OFFICIAL-DOC]Change Data Capture with Debezium & Outbox Event Ordering Specification— TinyCTO Architectural Standards