Transactional Outbox & Change Data Capture (CDC) Architecture

Executive Summary

In microservice architectures, updating a local database and publishing a corresponding event to an event broker is an error-prone distributed operation. If the application crashes between the database commit and the broker produce call, data corruption occurs. The Transactional Outbox pattern combined with Change Data Capture (CDC) mathematically eliminates this "dual-write" failure mode without expensive two-phase commit (2PC) protocols.

1. The Dual-Write Vulnerability

Consider the naive implementation:

BEGIN TRANSACTION;
  UPDATE accounts SET balance = balance - 100 WHERE id = 42;
COMMIT;
kafkaProducer.send("AccountDebited", account); // FAILS IF BROKER UNREACHABLE OR PROCESS CRASHES

The database update succeeded, but the event was never published. Alternatively, if the message is sent before committing, a database rollback leaves an orphaned event in Kafka.

2. PostgreSQL Logical Decoding & Debezium Pipeline

1. Atomic Local Outbox Insertion: The outbox record is inserted in the exact same relational transaction as the business entity.
2. PostgreSQL WAL Streaming: The PostgreSQL `pgoutput` plugin decodes mutations from the Write-Ahead Log into JSON/Avro structures.
3. Debezium Offset Tracking: Debezium commits Kafka message offsets only after receiving broker acknowledgments, guaranteeing at-least-once delivery with zero data loss.
4. Outbox Partition Routing: The event payload defines the Kafka destination topic and partition key directly from relational columns.