Thundering herd

A thundering herd occurs when many actors become ready at the same time and hit the same constrained resource.

Common triggers include cache expiry, service recovery, scheduled work, reconnect loops, and release of blocked workers.

flowchart LR
    Trigger[Shared trigger] --> A[Client A]
    Trigger --> B[Client B]
    Trigger --> C[Client C]
    Trigger --> D[Client D]
    A --> Dependency[Shared dependency]
    B --> Dependency
    C --> Dependency
    D --> Dependency

Symptoms

Load arrives as a sharp spike. Latency rises, queues grow, timeouts increase, and the dependency can fail again immediately after recovery.

Detection

Correlate request volume, cache misses, retries, queue depth, and dependency saturation around the same timestamp.

Mitigation

Spread work over time with jitter, request coalescing, staggered expiry, bounded concurrency, admission control, or single-flight behavior for shared cache fills.

Remove synchronization between clients

Jitter and staggered expiry are effective because they prevent many independent clients from making the same timing decision at once.

Trade-offs

Smoothing load can add latency for some callers. Coalescing requires care when requests are not truly equivalent.

Sources

• Google. Site Reliability Engineering, “Addressing Cascading Failures.”

Related knowledge

Retries and exponential backoffRetry selected transient failures with bounded attempts, increasing delays, and jitter so recovery attempts do not become new load spikes.

Load shedding and backpressureKeep overload bounded by slowing producers, limiting admitted work, or rejecting work before saturated components collapse.

TimeoutsBound how long work may wait for a dependency so stalled operations do not consume resources indefinitely.

CachingReuse a previously computed or fetched result to avoid repeating expensive work, at the cost of managing staleness and invalidation.