Data liberation: Apache Kafka's native cluster mirroring | Red Hat Developer
Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 901 words · 3 segments analyzed
Apache Kafka excels at moving data within a cluster. Leaders replicate to followers, consumers pull from any replica, and the entire machinery runs with minimal operational overhead. Moving data between clusters has never been that simple.Organizations run multiple Kafka clusters for good reasons: Geographic distribution, compliance boundaries, team isolation, version segregation. But once data lands in a cluster, getting a faithful copy into another has always required external tooling, careful coordination, and a healthy tolerance for operational surprises.KIP-1279 changes this. Cluster mirroring embeds cross-cluster replication directly into the Kafka broker. No external processes, no offset translation tables, no recompression. A destination broker fetches committed records from a source cluster using the same fetch protocol that followers already use, and appends them to local partition logs byte for byte. The result is a mirror that preserves offsets, compression, and consumer group state, making failover as simple as stopping the mirror and redirecting clients.This article covers the high-level architecture behind cluster mirroring, the state machine that governs mirror partitions, and the consistency guarantees that hold it all together. I then walk through 2 practical scenarios (disaster recovery and cluster migration), and end with a video demo.A new approach cross-cluster replicationMirrorMaker 2 (MM2) has served as the standard tool for cross-cluster replication since Kafka 2.4, running as a set of Kafka Connect workers that consume from a source cluster and produce to a destination cluster. Cluster mirroring takes a fundamentally different approach by embedding replication directly into the broker.Zero external infrastructure: Mirror fetcher threads run inside the broker process itself. There are no Connect workers to provision, monitor, or scale. A single CLI command (kafka-cluster-mirrors.sh --create) establishes a mirror; another (--start) begins replicating topics. The entire lifecycle is managed through the same Admin API used for topics and consumer groups.Byte-for-byte transfer: Compressed batches are replicated as raw bytes, never decompressing or recompressing them. A gzip, snappy, lz4, or zstd batch arrives at the destination in its original form. This eliminates the CPU overhead of a decompress/recompress round trip and preserves the producer's original compression choices.Exact offset preservation: The destination log maintains the same offsets as the source, including gaps left by topic compaction. Consumer groups fail over without offset translation: The committed offset on the source is the committed offset on the destination.One-command failover: Stopping a mirror (--stop) transitions partitions through a deterministic sequence: fetchers are removed, the last mirror epoch is persisted, the leader epoch is bumped, pending transactions are aborted, and a new control record expires all producer state. The partition becomes writable on the destination. No external coordination or offset queries required.Unclean leader election support: When the source cluster experiences an unclean leader election (ULE), the destination enters a recovery state. It waits for all assigned replicas, not just ISR members, to converge to the truncated offset before resuming replication. This ensures log consistency between clusters even when the source elects a leader with an incomplete log.The following table summarizes the key differences:AspectMirrorMaker 2Cluster mirroringDeploymentExternal Connect workersBroker embeddedCompressionDecompress and recompressByte-for-byte passthroughOffsetsLossy translation via topicIdentical across clustersConsumer failoverQuery offset sync topicDirect, no translationUnclean electionsNo handlingFull log convergenceSource compatibilityKafka 2.0+Kafka 2.1+MonitoringConnect specific toolingStandard broker JMX metricsWith cluster mirroring, destination brokers become active participants in cross-cluster replication. Each one fetches data directly from the source cluster using the standard fetch protocol and appends raw record batches to local partition logs.
Source and destination partitions share the same topic ID.Beyond data replication, the broker also handles metadata discovery, configuration syncing, groups offset syncing, and ACL propagation. Bandwidth control works on both sides. The destination broker enforces a configurable replication rate limit. On the source side, mirror fetch traffic presents as standard consumer requests, so existing client quota mechanisms apply without modification.ArchitectureThere are 3 main components collaborating within each destination broker. Figure 1 shows how they connect to each other and to the source cluster. MirrorMetadataManagerMirrorMetadataManager (MMM) is the orchestrator. Running on every broker, it implements the MetadataPublisher interface to react to changes in the KRaft metadata log. When the controller writes a MirrorTopicStateChangeRecord, the MMM on the affected partition's leader drives the corresponding state transition that triggers a specific operation (create, start, stop, pause, resume, recover, delete).MMM also maintains an Admin client connection to the source cluster.
Every 60 seconds by default, it refreshes source metadata: Discovering new topics that match configured include/exclude patterns, syncing topic configurations, fetching consumer group offsets, and validating that the source cluster ID has not changed. That last check prevents silent data corruption if someone accidentally reconfigures the mirror to point at a different cluster.ClusterMirrorCoordinatorClusterMirrorCoordinator (CMC) handles state persistence. It follows the same coordinator pattern used by the group coordinator and the transaction coordinator, managing shards of an internal compacted topic called __mirror_state (defaults: compact cleanup policy, 50 partitions, replication factor 3). Each mirror partition's state is stored as a key-value record in this topic, with optimistic concurrency control through leader epoch and state epoch fencing.MirrorFetcherThreadMirrorFetcherThread (MFT) does the heavy lifting. Extending Kafka's AbstractFetcherThread (the same base class used for intra-cluster replication), it fetches records from the source and appends them to local logs. Each thread maintains a dedicated NetworkClient with per-mirror authentication credentials, keeping SASL/SSL contexts isolated between mirrors. The fetcher manager keys threads by a three-dimensional identifier (fetcher ID, source broker endpoint, mirror name), enabling fine-grained load balancing and fast response to leader changes on the source.Mirror partition lifecycleA mirror partition progresses through a sequence of well-defined states.