Retry storm

A retry storm occurs when failing or slow requests trigger enough retries to create additional load on an already unhealthy dependency.

flowchart LR
    Requests[Original requests] --> Dependency[Slow or failing dependency]
    Dependency --> Errors[Timeouts and errors]
    Errors --> Retries[Client retries]
    Retries --> Dependency

The feedback loop can keep the dependency overloaded after the original fault starts to recover.

Symptoms

Request volume rises after errors begin. Recovery takes longer than expected, and several client layers can multiply one original request into many attempts.

Detection

Measure original requests separately from retry attempts. Inspect retry counts across every layer in the call path and correlate them with saturation and timeout rates.

Mitigation

Use bounded retries, exponential backoff, jitter, retry budgets, and explicit rules for which failures are safe to retry. Prefer one retry owner in a layered call path when possible.

Idempotency is required when a retry can repeat a side effect.

Retries can amplify an outage

Do not retry indefinitely or at every layer. A retry policy must bound attempts and load, and it must distinguish retryable failures from permanent or unsafe failures.

Trade-offs

Fewer retries can expose transient failures to callers. More retries can improve success during short faults but consume capacity needed for recovery.

Sources

• Amazon Web Services. “Timeouts, retries, and backoff with jitter.”

Related knowledge

Retries and exponential backoffRetry selected transient failures with bounded attempts, increasing delays, and jitter so recovery attempts do not become new load spikes.

TimeoutsBound how long work may wait for a dependency so stalled operations do not consume resources indefinitely.

IdempotencyMake repeated delivery of the same logical operation safe when retries, duplicates, or uncertain responses are possible.

Circuit breakersStop repeated calls to a dependency that is likely to fail, then probe for recovery before normal traffic resumes.

Load shedding and backpressureKeep overload bounded by slowing producers, limiting admitted work, or rejecting work before saturated components collapse.