Skip to content
HN On Hacker News ↗

How Uber Protects Against Retry Storms

▲ 102 points 44 comments by iscmt 14h ago HN discussion ↗

Pangram verdict · v3.3

We believe this text is mainly human-written, with some AI content.

6 %

AI likelihood · overall

Human
94% human-written 6% AI-generated
SEGMENTS · HUMAN 1 of 2
SEGMENTS · AI 0 of 2
WORD COUNT 194
PEAK AI % 56% · §2
Analyzed
Sep 17
backend: pangram/v3.3
Segments scanned
2 windows
avg 97 words each
Distribution
94 / 6%
human / AI fraction
Verdict
Human
Pangram v3.3

Article text · 194 words · 2 segments analyzed

Human AI-generated
§1 Human · 22%

IntroductionRetry storms historically impact business operations and brand trust. While retry configuration tuning and retry budgets provide meaningful mitigation at the service level, they’re manually configured and lack visibility into cross-service amplification caused by deep dependency chains and fan-out patterns. As a result, it can be difficult to shield infrastructure against the domino effect triggered by a single service outage deeper in the stack. A key reason is that retry behavior today isn’t context-aware. While we can control how many retries occur, we can’t precisely control when they occur. This stems from the challenge of reliably distinguishing between errors generated by a service and those merely propagated through it.As a result, retries are applied uniformly rather than conditionally.This approach works for transient or low-rate failures.

§2 Mixed · 56%

However, during moderate or severe degradation, it becomes counterproductive. Aggressively retrying against an already struggling service increases load, accelerates failure, and amplifies retry traffic across upstream dependencies. What begins as a localized outage can quickly escalate into a stack-wide incident—ultimately degrading, or in the worst case, completely breaking, the end user experience.One might argue that error codes from downstream services could be translated upstream to provide context for retries.