distributed systems / LEVEL 3

The retry storm after the timeout

EST.40 MIN

THEORY / RETRIEVAL

What to restore

Keep all attempts inside one end-to-end deadline
Use capped exponential backoff with full jitter
Retry only operations with an explicit idempotency contract

Every retry spends the caller's budget

A timeout per attempt is not an end-to-end deadline. Queueing, connection setup, execution and backoff all consume the same caller budget. Starting an attempt that cannot finish before that deadline increases load while its result is already useless.

Jitter breaks synchronized recovery

Capped exponential backoff limits retry frequency; full jitter chooses a value between zero and the current cap so many clients do not wake together. Retries still require idempotency, bounded attempts and overload-aware server signals.

CHECKPOINT

A request has 800 ms left and the downstream p99 is 700 ms. What must happen before scheduling another retry after a 200 ms backoff?

Confidence: 3/5

02 / IMPLEMENTATION

Implement the contract

Implement a deterministic full-jitter retry schedule. Each entropy sample is in 0..=1000 and selects that fraction of the current exponential cap. Clamp hostile samples, stop before exceeding the total backoff budget, and never overflow.

Initializing editor…

CLOUD SANDBOXnetwork off · 256 MB · 12 s

1 / 64 KB

OUTPUT

Runner is waiting for a submission.