Log-Centric Streaming: Partitioning, Zero-Copy I/O & Consumer Group Protocols

Executive Summary

Modern distributed streaming platforms discard volatile in-memory queues in favor of persistent, append-only commit logs. By modeling data streams as immutable, totally ordered sequences of events written sequentially to disk, systems like Apache Kafka and Redpanda achieve millions of messages per second with deterministic replayability.

1. The Zero-Copy Kernel Transfer Advantage

Traditional message brokers transfer data by copying bytes through multiple user-space buffers: $$\text{Disk} \rightarrow \text{OS PageCache} \rightarrow \text{App User Buffer} \rightarrow \text{Socket Buffer} \rightarrow \text{NIC}$$ This involves 4 context switches and 3 redundant memory copies.

Kafka utilizes the Linux `sendfile()` system call (Zero-Copy), bypassing user-space completely:

Result: CPU overhead drops by 80%, and network egress saturates 100 Gbps Ethernet interfaces directly from OS disk cache.

2. Partition Key Hashing & Total Ordering

A Kafka topic is composed of $P$ physical partitions. Strict total ordering is guaranteed only within a single partition. Messages with identical partition keys are deterministically mapped to the same partition using MurmurHash2: $$\text{Partition ID} = |\text{MurmurHash2}(\text{key})| \pmod P$$

3. Cooperative Sticky Consumer Group Rebalance

Legacy consumer group rebalance protocols (Eager Rebalance) forced all consumers in a group to revoke all assigned partitions before reallocating, causing massive latency spikes (stop-the-world rebalance storms). The Cooperative Sticky Assignor protocol resolves this:

1. Revokes only the specific partitions that must migrate to new consumers.
2. Unaffected consumers continue processing their existing partitions without interruption.