Client-Side Load Balancing at a Million Requests Per Second
Pangram verdict · v3.3
We believe that this document is a mix of AI-generated, AI-assisted, and human-written content
AI likelihood · overall
MixedArticle text · 1,749 words · 5 segments analyzed
Our busiest API ran its high-volume internal traffic through the cluster's shared edge ingress load balancer. For years we could never be sure whether a latency spike came from our own code or from reusing that shared edge router internally.In a previous post, we described how we built Zalando's Product Read API (PRAPI), serving millions of requests per second with single-digit-millisecond latency across 25 European markets. Every product page, search result, and checkout depends on it. A brief degradation has measurable impact on sales, resulting in high performance and availability requirements. The low latency is achieved through consistent-hash routing: Skipper, the cluster's edge load balancer, routes the same product ID to the same pod(s), helping to leverage pod-local caches in the underlying application. The routing infrastructure for this API matters.On launch, Skipper handled both edge routing and the internal traffic between our batching and single-get components. It was always my intention that client-side load balancing (CSLB) would replace the latter, and I had hoped it would be a fast-follow. But Skipper was fast, adding only a couple of hundred microseconds to each request, and the team was understandably reluctant to introduce significant change to a working system. Over the years, as incidents accumulated where the root cause was never quite clear (Skipper, or PRAPI?), it became harder to ignore the structural problem. For a single batch-of-100 request, PRAPI had a 100x exposure to Skipper. When Skipper sneezed, PRAPI got the flu.Some of those incidents, it would later turn out, were neither Skipper nor PRAPI. But we had no way to see that until we owned the routing decision, and the detailed logs that came with it.Skipper and the Fan-Out ProblemSkipper is Zalando's open-source Kubernetes ingress controller and HTTP router. It handles edge load balancing brilliantly: consistent-hash routing, bounded load protection, fade-in for new pods. We contributed key features to Skipper ourselves, including minimising cache loss during scaling and preventing overload from hyped products. We still use Skipper for all single-product GET requests today.The problem was our batch endpoint. PRAPI's product-sets component unpacks a single batch request into up to 100 parallel downstream calls to individual products pods.
Each of those 100 calls transits Skipper. Skipper adds only a couple of hundred microseconds per hop, but a batch waits on up to a hundred of those hops at once, so its latency tracks the slowest of the hundred, not the typical one. And Skipper is shared infrastructure: we run on the same fleet as the rest of our cluster, on a global configuration we inherit rather than set.Product-sets fan-out through the ingress load balancerDuring incidents, we could never be certain whether latency spikes originated in Skipper or in our own code. It sat in the hot path of every request, we did not run it, and we could not cleanly separate its behaviour from ours. Even when Skipper was fast, that shared fate was the problem.We decided that for high fan-out internal traffic, the routing decision should live inside the calling process itself. Edge traffic, where Skipper excels, should stay exactly where it is. We were not replacing Skipper; we were graduating the internal fan-out path to a client-side load balancer that runs inside the process.Product-sets routing directly to products podsBuilding the Same Hash RingWe did not need client-side balancing in the abstract, we needed the exact same ring as Skipper, in our own process. The most critical constraint was therefore hash parity. During migration, both Skipper and our client-side load balancer would route requests to the same pool of products pods. If the hash rings disagreed, a product that Skipper routes to pod A might be routed to pod B by our library. That would split caches and double DynamoDB load, exactly the opposite of what we wanted.We implemented the same algorithm Skipper uses: xxHash64 on a configurable virtual-node ring. Each endpoint URL is placed at 100 positions on a 64-bit hash ring, matching Skipper's default. When a request comes in, the product ID is hashed and a binary search on the ring finds the nearest endpoint clockwise.This means that adding or removing an endpoint remaps only about 1/N of keys, minimising cache churn. And because both Skipper and our library use the same hash function and the same number of virtual nodes, they produce identical rings for the same set of pods.
A bank of unit tests pins this down: they assert our ring places the same keys on the same endpoints as Skipper's algorithm for any pod set, and run on every build, so a later change cannot silently drift from Skipper. We confirmed it held in production too, during the canary: cache hit ratios stayed identical on both paths.We wrote it as a standalone, framework-free JVM module with the long-term intention of lifting it out of this service. Its only real dependency is a small zero-allocation hashing library, for the xxHash64 that matches Skipper; everything else, the ring, the occupancy accounting, the bounded-load math, is the JDK standard library. The Kubernetes client and Micrometer sit at the edges, for discovery and metrics.Kubernetes DiscoveryOur load balancer needs to know which pods exist. The initial approach was polling the Kubernetes EndpointSlice API every few seconds, but polling was a pattern we had learned to treat with respect. Zalando had previously experienced incidents where Akka Cluster deployments polling the Kubernetes API at high frequency had taken out the control plane entirely. PRAPI runs on hundreds of product-sets pods; hundreds of pods polling independently, each on their own schedule, is exactly the kind of aggregate that causes those incidents.We switched to a watch-based Kubernetes informer. On startup it lists the current EndpointSlices to seed the ring, then holds a persistent watch that streams changes in real time. A 2-second debounce coalesces rapid changes during scale-ups into a single ring update, preventing the ring from rebuilding once per pod.informer (list + watch) |-- on startup: list current EndpointSlices, seed the ring +-- then stream add / update / delete events on each event: |-- update the local per-slice cache +-- schedule applyLocalState() with a 2s debounce +-- the debounce folds a rolling deploy's churn into a single ring update If the Kubernetes API becomes unavailable, the last good endpoint set remains in place. The load balancer never presents an empty ring due to a transient API blip. Staleness is handled by connection errors at the HTTP layer and the caller's retry logic.Fixing the Pipeline FirstOne thing had to come first, because none of what follows was possible until we fixed it. Every fade-in curve, load-signal experiment, and zone trial needed many small, fast deployments.
On the pipeline we started with, we could manage one a day at best, and even that often tripped alarms on the way out.The pipeline had degraded over years of accumulated caution. Every manual approval gate and 180-second sleep had been added in response to some past incident, until the cumulative result was slower and more fragile than the system it protected. Builds took 21 minutes. A single feature deploy meant babysitting traffic steps for an entire workday; the median run took 4 hours and 49 minutes, and the worst on record was 4 days and 21 hours.The fix was three PRs. Build caching cut wall time from 21 minutes to 12. We collapsed 40-plus manual traffic steps into a single CI/CD step. The approval gates came out, replaced by a sequenced market-group rollout (test -> eu-0 -> eu-1 -> eu-2) that uses the smaller regions as an alarm buffer before the critical eu-2 region. Median deployment time fell from 289 minutes to 128.CI/CD pipeline run durations, down from a worst case of nearly five days to an hour or twoOnce the team trusted the pipeline, the pace changed: small, reversible steps replaced big, risky batches. And it was not just mine to use: within weeks other engineers were running their own experiments in parallel, on cache tuning and beyond, because the cost of trying something had collapsed. Everything that follows in this post was built on that velocity.Rolling It Out SafelyWe built three toggles for the rollout:CSLB_ENABLED: a boolean off switchCSLB_PERCENTAGE: a 0-100 traffic ramp controlling what fraction of requests use client-side routing vs the Skipper fallbackImplicit Skipper fallback: any request not routed client-side, or any client-side routing failure, falls through to Skipper transparentlyWe rolled out incrementally: 1% in our canary market group, then 10%, 50%, and finally 100% across all market groups. At each step we compared latency, error rates, and cache hit ratios between the two paths.The effect was immediate. Latency dropped. Spikes that we had attributed to "the network" or "something in the stack" disappeared. At peak, over a million requests per second were re-routed off Skipper.
Requests per second through Skipper: peaks approaching a million collapse to near zero after the CSLB cutoverproduct-offers latency before and after CSLB: daily spikes flatten to a steady baselineAt 100%, the Skipper fallback path still exists in code but is never hit. It remains as an emergency off switch: one ConfigMap change (rolled out via blue-green deployment) and all traffic flows through Skipper again.The immediate win was stability. But removing a million requests per second from shared infrastructure had a side effect we had not anticipated: Skipper's fleet for our routes scaled down from over 50 pods to a minimum of 8. What started as a latency and operability project had become a cost project. And owning the load balancer meant we could keep going.Daily Skipper node-group cost falling from around $450 to roughly $110 a day after the CSLB cutoverEliminating Scale-Up Spikes: N-Ring Fade-InNow we could ship and measure quickly. The first target was scale-up spikes, long accepted as an unfortunate constant. The team was cautious about autoscaler sensitivity for good reason. Although we had taken Skipper out of the fan-out path, the cold-cache problem on scale-up remained.When the Horizontal Pod Autoscaler adds 50 new pods, a naive implementation routes their full share of traffic to them immediately. Those pods have cold caches, so all fifty miss into DynamoDB at once, and the read burst spikes latency across the fleet.Skipper has a fade-in mechanism to address this: a probabilistic pre-filter that runs before the consistent-hash ring and adds or removes new pods during fade. While this works well, it often means that new pods are warmed with products they may not serve once the fade completes.Our optimisation is N-ring fade-in. Each scale event creates a new ring that is a superset of the established ring.