Skip to main content

> STREAMING_LOGS // Quorum // CP

Multi-Tenant Segmented Storage Streaming

Decoupled compute-and-storage streaming architecture utilizing Apache Pulsar brokers and Apache BookKeeper ledgers for infinite retention and millions of topics.

OverviewCAP & PACELCArchitectures18Topology SizerInteractiveManuals10Comparison Matrix22
Back to Architecture Catalog
CAP: CPPACELC: PC/ECConsensus: Quorum

Problem Statement & Architectural Hypothesis

Monolithic brokers couple partition storage with broker compute, forcing expensive cluster rebalances and partition migrations whenever disk limits are approached.

Formal Distributed Guarantees

  • ⚡Instant partition rebalancing (BookKeeper segment handoff)
  • ⚡Native multi-tenancy with strict tenant rate quotas
  • ⚡Native tiered storage streaming direct to Parquet/Iceberg

Handled Failure Modes

DS-FAIL-03: Unbounded Consumer Lag Spiral
DS-FAIL-12: Hot Partition Skew
Raw Inspection & ExportView Raw Markdown

3 Maturity & Scale Configurations

Step-by-step production configurations from single-cluster baseline up to multi-datacenter ultra-scale.

INITIAL TIER
Throughput Target:

20,000 msg/sec

p99 Latency:

< 25ms

Delivery Guarantee:

At-Least-Once Delivery

Topology:

3 broker pods + 3 BookKeeper bookie pods.

Stack Components:
Apache PulsarApache BookKeeperZooKeeper / KRaft
⚠️ Operational Tradeoff: Dual-layer operations (managing both brokers and bookies).
SCALED TIER
Throughput Target:

350,000 msg/sec

p99 Latency:

< 6ms

Delivery Guarantee:

Strict Deduplication (Broker Deduplication Engine)

Topology:

6 stateless brokers + 12 bookies with dedicated journal and ledger NVMe disks.

Stack Components:
Apache PulsarApache BookKeeperPulsar FunctionsGrafana
⚠️ Operational Tradeoff: Higher component operational complexity.
ULTRA_SCALE TIERMISSION CRITICAL
Throughput Target:

2,000,000 msg/sec

p99 Latency:

< 3.2ms

Delivery Guarantee:

Global Geo-Replication across 3 Continents with Automatic Failover

Topology:

Multi-datacenter mesh across US, EU, and APAC with asynchronous geo-replication.

Stack Components:
Pulsar MeshBookKeeper AutorecoveryApache Iceberg SinkS3 Tiered Storage
⚠️ Operational Tradeoff: Cross-continental network latency on geo-replicated namespaces.

Infrastructure as Code: Terraform, Kubernetes & Engine Configs

Production-ready automation manifests ready for deployment on Kubernetes and cloud providers.

Terraform (HCL)main.tf
resource "helm_release" "pulsar" {
  name       = "pulsar"
  repository = "https://pulsar.apache.org/charts"
  chart      = "pulsar"
  version    = "3.3.x"

  values = [
    file("${path.module}/pulsar-production-values.yaml")
  ]
}
Kubernetes (YAML)k8s-manifest.yaml
apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: pulsar-mesh
spec:
  broker:
    replicas: 6
  bookkeeper:
    replicas: 12
    journal:
      volumeSize: 200Gi
    ledgers:
      volumeSize: 1000Gi
Engine Configurationconfig.properties
managedLedgerDefaultEnsembleSize=3
managedLedgerDefaultWriteQuorum=3
managedLedgerDefaultAckQuorum=2
brokerDeduplicationEnabled=true
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=tinycto-pulsar-offload
AI Summary — Multi-Tenant Segmented Storage Streaming
AEO / GEO / Perplexity Indexable

Decoupled compute-and-storage streaming architecture utilizing Apache Pulsar brokers and Apache BookKeeper ledgers for infinite retention and millions of topics.

CAP & PACELC TheoremsCAP: CP // PACELC: PC/EC
Consensus ProtocolQuorum
Ultra-Scale Target2,000,000 msg/sec (< 3.2ms)
Handled Failure ModesDS-FAIL-03: Unbounded Consumer Lag Spiral; DS-FAIL-12: Hot Partition Skew

Architecture Blueprint FAQs

What is the mathematical CAP and PACELC classification of Multi-Tenant Segmented Storage Streaming?

Multi-Tenant Segmented Storage Streaming is classified under CAP as CP and under PACELC as PC/EC. During network partitions, it prioritizes consistency, maintaining strict state guarantees.

How does the Quorum consensus protocol operate in this architecture?

This blueprint relies on Quorum for quorum-based state machine replication. Leader election, log compaction, and split-brain prevention are enforced through monotonic terms and fencing tokens.

Which distributed failure modes does this architecture handle?

The architecture explicitly handles the following failure modes: DS-FAIL-03: Unbounded Consumer Lag Spiral, DS-FAIL-12: Hot Partition Skew, ensuring no silent divergence or message loss.

What are the throughput and latency differentials between Initial and Ultra-Scale tiers?

The Initial tier targets 20,000 msg/sec with < 25ms p99 latency (3 broker pods + 3 BookKeeper bookie pods.), whereas Ultra-Scale scales to 2,000,000 msg/sec with < 3.2ms (Multi-datacenter mesh across US, EU, and APAC with asynchronous geo-replication.) using: Pulsar Mesh, BookKeeper Autorecovery, Apache Iceberg Sink, S3 Tiered Storage.

How is this architecture provisioned via declarative Infrastructure as Code?

The provided Terraform HCL, Kubernetes manifest, and engine configuration properties furnish immediate production templates for Kubernetes clusters and event broker topologies.