GitHub - telemetry-sh/retry-storm-lab: Simulate how retry policies amplify traffic during partial outages.

A retry is not free. See when your recovery policy becomes the outage. Retry Storm Lab is a deterministic Python simulator for partial dependency failures. It models fresh requests and retries competing for the same capacity, then compares immediate, fixed, exponential, and full-jitter backoff policies.

Why this exists Retries improve reliability when failures are brief and spare capacity exists. During a sustained partial outage, synchronized clients can instead amplify demand, starve fresh requests, and keep a recovering service overloaded. The simulator makes that feedback loop visible:

Generate a steady stream of fresh requests. Reduce successful capacity during a configurable failure window. Reschedule failed and over-capacity attempts using the selected policy. Measure amplification, peak demand, abandoned requests, and recovery time. Compare all four policies against the exact same scenario and random seed.

This is an intentionally small queueing model, not a production capacity planner. It omits network latency distributions, client timeouts, autoscaling, circuit breakers, and correlated downstream failures. Its job is to make retry dynamics easy to inspect and discuss. Run the interactive lab Python 3.11 or newer is required. python -m venv .venv source .venv/bin/activate pip install -e . streamlit run app.py Open http://localhost:8501, choose a preset, and adjust the workload from the sidebar. Use the CLI retry-storm-lab --preset "Rate-limit spiral" --compare retry-storm-lab --preset "Slow recovery" --policy full_jitter --json Example comparison: immediate amplification=1.11x peak=2720 rps success=97.2% recovery=1s fixed amplification=1.08x peak=1600 rps success=99.9% recovery=3s exponential amplification=1.07x peak=1600 rps success=100.0% recovery=3s full_jitter amplification=1.10x peak=1651 rps success=99.9% recovery=3s

Exact values depend on the selected preset and seed.

Use the simulation engine from retry_storm_lab import BackoffPolicy, SimulationConfig, simulate

result = simulate( SimulationConfig( baseline_rps=800, server_capacity_rps=1_000, outage_failure_percent=80, policy=BackoffPolicy.FULL_JITTER, ) )

print(result.amplification_factor) print(result.peak_rps) print(result.recovery_seconds) The engine uses aggregate 100 ms buckets. Full jitter distributes retries individually with a seeded pseudo-random generator, so the same configuration always produces the same result. Optional aggregate telemetry The lab works entirely offline. To record aggregate usage bands with Telemetry, install the optional dependency and provide a project API key: pip install -e '.[telemetry]' export TELEMETRY_API_KEY=your_project_api_key streamlit run app.py Only the selected policy, broad capacity/amplification bands, and whether the run recovered are sent. Raw workload values and timeline data are not logged. Development python -m unittest discover -s tests -v python -m compileall app.py retry_storm_lab License MIT