Pangram verdict · v3.3
We believe that this document is a mix of AI-generated, AI-assisted, and human-written content
AI likelihood · overall
MixedArticle text · 1,437 words · 6 segments analyzed
Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat them as one thing. Failover becomes a runbook: restore, repoint DNS, and wait for an outage that, on paper, you’d already paid to survive.Linkerd’s multicluster extension closes that gap by letting several clusters present a service as a single, load-balanced endpoint. The part that the official tasks gloss over is that a real platform almost never picks one multicluster mode. Some services want federation (same service everywhere, one endpoint, automatic failover). While others want mirroring (reach a specific remote service by name). And you frequently want both patterns living on the same set of links. The docs walk through each mode on its own. This post wires all three together across three GKE clusters, with a full-mesh link topology, a chaos test that takes out an entire cluster, and scripts you can clone and run on a fresh GCP project.Companion repo: Every script referenced here lives in this repository. Feel free to clone it, set your project ID, and run it.Linkerd multicluster modes: Gateway, flat, and federatedLinkerd’s multicluster extension supports three modes. The nice thing is they’re not mutually exclusive: on the same set of linked clusters, the mode is chosen per service via a label.
ModeLabelWhat happensNetwork RequirementHierarchical (gateway)mirror.linkerd.io/exported=trueService mirrored as <svc>-<cluster>, traffic routed through a gatewayGateway IP reachableFlat (pod-to-pod)mirror.linkerd.io/exported=remote-discoveryService mirrored as <svc>-<cluster>, traffic goes directly to remote podsFlat network (pod IPs routable)Federatedmirror.linkerd.io/federated=memberAll same-name services unioned into <svc>-federated, load balanced across all clustersFlat network (pod IPs routable)The distinction that matters operationally is that hierarchical mirroring works on any network. Only the gateway IP needs to be reachable, while flat and federated modes need real pod-to-pod connectivity. On GCP, VPC-native GKE clusters on peered VPCs give you that flat network for free. So, you can run federated services for your core workloads over a flat network and still mirror a specialized service through a gateway from a cluster that isn’t on that network. Most platform teams I’ve seen end up with exactly this kind of mix.Multi-region architecture: GKE cluster setupWe have three GKE clusters across three regions, fully linked to each other (six directional links total). Three demo services, each using a different multicluster mode:frontend is federated and runs in all three clusters. A single federated frontend service in each cluster load-balances across all nine pods (3 replicas × 3 clusters). When a cluster goes down, the remaining six pods absorb the traffic with no application changes.api is flat-mirrored and runs in west and east. The north cluster consumes it as api-west and api-east, which are explicit remote service names with traffic sent straight to the remote pods. This is what you reach for when the client needs to decide which backend it talks to, for example, to keep a request in-region for data locality.analytics is gateway-mirrored and runs only in east. Exported through the Linkerd gateway so west and north reach it as analytics-east-gw without needing flat-network connectivity to east’s pods.
It’s here mainly to prove that gateway mode coexists with flat and federated modes on the same links.Deployment prerequisites: GKE, Linkerd, and CLI toolsA GCP account (free-tier credits cover this. Use three standard clusters with small node pools)gcloud CLI, authenticated (gcloud auth login)kubectl v1.28+step CLI, brew install step (for certificate generation)helm v3~30 minutes for the full setupThe infra script enables the compute and container APIs for you, so a brand-new project works out of the box.Step 0: ConfigureClone the repo, create a local .env file from the example file, and customize it for your GCP project. The defaults are enough for the rest of the demo, so in most cases you only need to change the project ID.git clone <your-repo-url> cd blog-linkerd-federation cp env.example .env Open .env and set at least your project ID. The file ships with sensible defaults for everything else:export GCP_PROJECT="your-project-id"
export REGION_WEST="us-central1" export REGION_EAST="us-east1" export REGION_NORTH="europe-west1"
# One zone per region. We pin node-locations to a single zone so num-nodes is # the TOTAL node count — see the cost note below for why this matters. export ZONE_WEST="us-central1-a" export ZONE_EAST="us-east1-b" export ZONE_NORTH="europe-west1-b"
export CLUSTER_MACHINE_TYPE="e2-medium" export CLUSTER_NODE_COUNT="1" export FRONTEND_REPLICAS="3" At minimum, set GCP_PROJECT. Everything else ships with sensible defaults: three regions, one zone per region, and small node pools to keep the cost down. If you run cat .env, you should see the full set of variables populated.Load the variables into your current shell so the scripts can read them:source .env Every script below reads from this file, and they all run with set -euo pipefail, so a missing variable fails loudly rather than silently.
That’s why env.example carries the full set, the VPC and cluster names included, instead of just the project ID.Step 1: Provision three GKE clusters with VPC peeringRun the infrastructure script to create the networks and clusters. This takes about 10–15 minutes, so it’s a good point to grab a coffee../scripts/01-infra.sh This script does the following:Enables the compute and container APIs (no-op if they’re already on).Creates three VPCs with non-overlapping pod and service CIDRs, a hard requirement for VPC peering.Sets up full-mesh VPC peering (west↔east, east↔north, north↔west) with --export-custom-routes and --import-custom-routes so pod CIDRs are actually advertised. This is what gives us the flat network.Creates three GKE Standard clusters, one per VPC/region, each pinned to a single zone.Renames the kubectl contexts to west, east, north.Here’s the address plan the script uses. The ranges are intentionally non-overlapping so VPC peering can route pod traffic correctly:ClusterVPC SubnetPod CIDRService CIDRwest10.10.0.0/2010.100.0.0/1410.104.0.0/20east10.20.0.0/2010.108.0.0/1410.112.0.0/20north10.30.0.0/2010.116.0.0/1410.120.0.0/20Non-overlapping ranges are non-negotiable. If pod CIDRs overlap across peered VPCs, routing breaks silently. Pods get responses from the wrong cluster, or connections time out with nothing useful in the logs. Ask me how I know.One zone, not three. A GKE regional cluster places --num-nodes nodes in each of three zones by default. With --num-nodes 1 that’s 3 nodes per cluster, 9 total, and triple the bill.
The script pins --node-locations to a single zone so CLUSTER_NODE_COUNT=1 really means one node per cluster.Cost note: Three Standard clusters with one e2-medium node each run roughly $10–15/day total for this demo (management fee + nodes + a small gateway load balancer on east). The teardown script removes everything.Install Linkerd into all three clusters using a shared trust anchor. The script generates the certificates, installs the control plane, and configures each cluster to trust the others for cross-cluster mTLS../scripts/02-linkerd-install.sh This generates a root CA and per-cluster issuer certificates, then installs Linkerd on all three clusters:root.crt (shared trust anchor) ├── issuer-west.crt + issuer-west.key ├── issuer-east.crt + issuer-east.key └── issuer-north.crt + issuer-north.key Per-cluster issuer certs are a production habit worth keeping: if one cluster’s issuer is compromised you rotate it in isolation, without touching the others. The shared root is what lets cross-cluster mTLS work at all. Every proxy can verify every other proxy’s identity back to the same anchor.To keep resource usage (and cost) down, this installs the control plane only with no Viz add-on.Step 3: Install multicluster and create a full-mesh link topologySet up the multicluster components and create a full-mesh topology between the clusters.
After this step, every cluster can consume services from every other cluster../scripts/03-multicluster-setup.sh This is the step with the most going on. We create six directional links, every cluster linked to every other cluster, so every cluster gets a <svc>-federated service for federated workloads, and every cluster can consume mirrored services from any other.The wrinkle is the gateway. Only east needs one (it’s the only cluster exporting analytics hierarchically), so we enable the gateway in east’s install and leave everyone else gatewayless. One install per cluster, all flags at once, no re-running install a second time to bolt a gateway on afterward:# west: gatewayless, with one controller per cluster it consumes from linkerd --context west multicluster install --gateway=false \ --set controllers[0].link.ref.name=east \ --set controllers[1].link.ref.name=north \ --set controllers[2].link.ref.name=east-gw \ | kubectl --context west apply -f -
# east: gateway enabled here, controllers for the clusters it consumes linkerd --context east multicluster install --gateway=true \ --set controllers[0].link.ref.name=west \ --set controllers[1].link.ref.name=north \ | kubectl --context east apply -f -
# north: gatewayless, controllers for west, east, and east's gateway link linkerd --context north multicluster install --gateway=false \ --set controllers[0].link.ref.name=west \ --set controllers[1].link.ref.name=east \ --set controllers[2].link.ref.name=east-gw \ | kubectl --context north apply -f - Note the controller count. The service-mirror controller runs on the consuming side, one per link. west and north each consume analytics via the gateway, so they get a third controller for the east-gw link; east doesn’t consume its own analytics, so it only needs two.