Your Retry Policy Is a DDoS You Wrote Yourself

Your Retry Policy Is a DDoS You Wrote Yourself

On August 17, 2026, GitHub went down for seven and a half hours. The code that kept it down was the code meant to bring it back up: retries, amplifying a failure tenfold. Here is the math of retry storms, and the three controls that stop them.

At 13:40 UTC on August 17, 2026, GitHub started to fall over. It stayed down for seven hours and thirty-five minutes. The most expensive part of the outage was not the part that started it. It was the recovery code.

One component hit a concurrency limit and stopped scaling. Requests timed out. Every client that timed out did the sensible, responsible, catastrophic thing: it retried. Retries piled onto a saturated system, creating more timeouts, creating more retries. GitHub’s availability report: “A latent client retry bug sharply amplified traffic to one internal authentication endpoint, which slowed recovery for the Copilot Token Service.” The numbers behind that sentence: a token service that normally handled 7,000 to 9,000 requests per second was hit with 70,000 to 100,000. Ten times the normal load, generated by GitHub’s own clients, at the exact moment the system needed less load to recover.

That incident is real, dated, and on the record. The hypothetical version runs in your infrastructure right now, waiting for its trigger.

The Two Ways Teams Get This Wrong

The first bad option is to never retry. A request fails and you surface the error immediately. Honest, simple, and wrong at scale. Transient failures are a fact of distributed systems: a packet drops, a connection resets, a load balancer drains a node mid-request. Zero retries converts every blip into a user-facing failure, and your on-call learns to dread deploy windows for no reason.

I understand the appeal. No retries means no retry storms, the way no cars means no traffic jams. For one class of operations it is even correct: anything non-idempotent, like charging a card, should not be blindly retried. But as a general policy it trades one failure mode for a worse user experience every single day.

The second bad option is to retry without bounds. Three attempts with a fixed one-second delay, a loop that retries until success, the SDK default you never looked at. This is where most teams live. It feels safe because each individual retry is reasonable. Nobody reviews a retry loop and thinks this is the outage. The retry is always somebody else’s DDoS.

The gap every explainer skips is the arithmetic between those two options. Retries do not add load. They multiply it. And once the multiplication starts, the system can stay broken after the original trigger is gone. That second fact turns a bad afternoon into a seven-hour outage, and almost nobody writes about it.

Do the Multiplication

Say your request path has three layers: your service, a gateway, an auth service, a database. Each layer retries a failed call three times. One user request fails at the database. The auth service retries three times. The gateway retries three times, and the auth service retries each of those three more times. Your service retries the gateway three times. One user request becomes 1 + 3 + 9 + 27 = 40 requests against a database that was already struggling. Retries multiply across layers, and nobody budgeted for the product of all of them.

Now add synchronization. A thousand clients fail at the same instant. Each waits exactly one second and retries, so a thousand retries arrive at the same instant, one second later. The dependency, which might have recovered in that second, is hit by the same synchronized wall that flattened it the first time. The thundering herd: not just more load, but load arriving in lockstep.

Now the cruelest part. Suppose the trigger clears: the bad deploy rolls back, the database recovers. The system does not recover with it. The backlog of retries generates enough load on its own to keep the dependency saturated. Timeouts continue, so retries continue, so saturation continues. The failure is self-sustaining, a metastable failure: a state the system stays in even though nothing is pushing it there anymore. You built a machine that sustains its own outage, and the only way out is shedding load by hand, usually at an hour when nobody wants to be doing it by hand.

The design choices are not accidents. A timeout converts slow into failed, which converts it into retried. Set it below the dependency’s p99 latency and you manufacture failures out of requests that would have succeeded, then multiply them. Retry a 429 or a 503 and you might get somewhere. Retry a 400 and you fail the same way every time, at full speed, forever.

Write Retries That Cannot Amplify

Here is the canonical shape of a safe retry. Three controls, each doing a different job.

First, exponential backoff with full jitter. Plain exponential backoff synchronizes your clients: they all fail together, wait the same 1, 2, 4, 8 seconds, and retry together. Jitter breaks the synchronization by randomizing the wait. Marc Brooker’s classic AWS Architecture Blog post tested the variants and found full jitter the best performer. Here it is in Python with tenacity:

from tenacity import retry, stop_after_attempt, wait_random_exponential, retry_if_exception_type

class TransientError(Exception):
    pass

@retry(
    stop=stop_after_attempt(4),
    wait=wait_random_exponential(multiplier=0.5, max=10),
    retry=retry_if_exception_type(TransientError),
    reraise=True,
)
def call_dependency(payload):
    response = http_post("https://deps.internal/charge-path", json=payload, timeout=2.0)
    if response.status_code in (429, 503):
        raise TransientError(f"retryable: {response.status_code}")
    response.raise_for_status()
    return response.json()

A few things are worth noting. First, the stop condition is absolute: four attempts, then the error propagates. Second, wait_random_exponential is full jitter with exponential growth, capped at ten seconds. The randomness is the point, not decoration. Third, the retry predicate only fires on 429 and 503. A 400 raises immediately. Retrying a request the server has told you is malformed is not resilience. It is a loop.

Second, a retry budget. Backoff bounds the wait; it does not bound the volume. If all of your traffic is failing, jittered retries still double your load against a dying dependency. The fix, borrowed from Finagle: retries may consume at most a fixed fraction of request volume, say 20 percent, and when the budget is empty, requests fail fast.

import threading, time

class RetryBudget:
    def __init__(self, ratio=0.2, window=10.0):
        self.ratio = ratio
        self.window = window
        self.lock = threading.Lock()
        self.tokens = 0.0
        self.last = time.monotonic()

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self.last
        self.last = now
        self.tokens = min(self.tokens + elapsed * self.ratio, self.window * self.ratio)

    def allow_retry(self):
        with self.lock:
            self._refill()
            if self.tokens >= 1.0:
                self.tokens -= 1.0
                return True
            return False

Walk through it: every second deposits 0.2 retry tokens and each retry spends one. When the dependency is healthy, retries are rare and the bucket stays full. When everything fails at once, the budget drains in seconds and further retries are refused, capping amplification near 1.2x. The refill is the elegant part: the budget recovers on its own as the failure rate drops. Fail fast is not giving up. It is load shedding with a receipt.

Third, a circuit breaker. The budget caps volume; the breaker stops calling a dependency that has proven it cannot answer. After a threshold of consecutive failures, the breaker opens and calls fail immediately for a cooldown. Then it lets one probe through. Probe succeeds, the breaker closes. Probe fails, it opens again.

import time

class CircuitBreaker:
    def __init__(self, threshold=5, cooldown=30.0):
        self.threshold = threshold
        self.cooldown = cooldown
        self.failures = 0
        self.opened_at = None

    def call(self, fn, *args, **kwargs):
        if self.opened_at is not None:
            if time.monotonic() - self.opened_at < self.cooldown:
                raise TransientError("circuit open: failing fast")
            result = fn(*args, **kwargs)  # half-open probe
            self.failures = 0
            self.opened_at = None
            return result
        try:
            result = fn(*args, **kwargs)
        except TransientError:
            self.failures += 1
            if self.failures >= self.threshold:
                self.opened_at = time.monotonic()
            raise
        self.failures = 0
        return result

The state machine is the whole point, documented as a stability pattern in Michael Nygard’s Release It! back in 2007. Closed means normal. Open means the dependency is presumed dead and nobody pays the cost of finding out again for thirty seconds. Half-open is the single probe testing whether the world changed. What the breaker buys you that backoff does not: backoff spaces out doomed requests, but the requests still go. The breaker stops them entirely, which is the only thing that lets a saturated dependency drain.

See It As a Pipeline

Here is the failure as a pipeline, with the three controls marked where they bite.

  client request
       |
       v
  +------------+    timeout fires     +----------------+
  | dependency | ------------------> | retry decision |
  |  saturated |    (slow -> failed)  +----------------+
  +------------+                             |
       ^                                     | allow?
       |                    +----------------+------------------+
       |                    | 1. backoff+jitter: desync the herd |
       |                    | 2. retry budget: cap volume at 20% |
       |                    | 3. circuit breaker: stop calling   |
       |                    +----------------+------------------+
       |                                     |
       +---- retries arrive in lockstep ------+
              (without controls: amplification,
               thundering herd, metastable stall)

Without the controls, this is positive feedback. With them, every stage fails toward less load. That direction is the whole design: under uncertainty, send less traffic, not more.

Configure It Where the Traffic Actually Flows

Code-level retries are half the story. In a service mesh, the proxy between your services has its own retry policy, and the default is rarely what you want. Here is an Istio VirtualService that retries the way the code above does: bounded attempts, a per-try timeout, retries only on the statuses that deserve them.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments
spec:
  hosts: [payments.internal]
  http:
  - route:
    - destination: { host: payments.internal }
    timeout: 8s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx,retriable-status-codes
      retryRemoteLocalities: true

The route timeout of eight seconds bounds the total time one request can spend, retries included. attempts: 3 is the absolute cap. perTryTimeout: 2s is the line most teams are missing: without it, one slow attempt eats the whole eight seconds. retryOn keeps retries to server errors, never 4xx. Know the product of every retry policy on a request path: the mesh retry and the code retry multiply if you configure both without thinking.

Then prove it in staging. Inject a delay into the dependency and watch the breaker open:

# add 3s latency to half the traffic, then watch retries stay flat
kubectl apply -f fault-injection-delay.yaml
kubectl logs -l app=payments-gateway --since=5m | grep -c "circuit open"

If the breaker never opens under injected failure, your resilience story is a story, not a system. Chaos engineering for retries is a one-afternoon exercise. It is the difference between believing your retry policy is bounded and knowing it.

Where This Breaks

No hedging. These controls have real limits.

• Retry budgets assume failures are roughly uniform. A poison-pill request consumes budget the same way a transient blip does. Budgets cap amplification; they do not classify failures.
• A breaker that opens too eagerly will reject traffic a healthy dependency could have served. Tune the threshold from production failure rates, not from a blog post.
• Retrying non-idempotent operations is still wrong with all three controls in place. A bounded, jittered, budgeted retry of “charge the card” is a bounded, jittered, budgeted double charge. Pair retries with idempotency keys, or do not retry.
• Timeouts are load-bearing. A timeout below p99 latency manufactures the failures your retries then multiply. Measure the latency distribution before you set any of these numbers.
• The controls compose across layers, and so does their absence. A budget at your service and no budget at the gateway still lets the gateway amplify. Audit the whole path.

Build It If / Skip It If

Build the full version if your service calls anything over a network in a request path that matters: backoff with full jitter, an absolute attempt cap, a retry budget around 10 to 20 percent, and a circuit breaker on every dependency with a measured failure rate. Full version: the three code controls, the mesh config to match, the fault-injection test in staging, and a dashboard of retry rate as a percentage of total requests. The first sign of a storm is that number climbing.

Skip it if the call is local and in-process, the operation is non-idempotent with no key, or a failed dependency makes the request meaningless anyway. A single-user CLI calling a local daemon does not need a circuit breaker. It needs an error message.

The minimal viable version fits in an afternoon:

1. Cap every retry loop at a fixed attempt count. Grep for unbounded retry loops and fix those first.
2. Replace fixed or plain exponential delays with full jitter. One helper function, used everywhere.
3. Retry only retryable statuses: timeouts, 429, 503. Everything else fails fast.
4. Add the breaker to the one dependency whose failure would hurt the most. Measure its failure rate for a week, then tune.

Stop writing retries that assume the dependency will be there when you ask again. Bound them, desynchronize them, budget them, and put a breaker behind them. Then go inject a failure in staging and watch the system do the boring, correct thing.

What is the worst retry storm you have seen, or the most creative retry loop you have found in a codebase? I am collecting specimens.

Resources

1. GitHub availability report: August 2026 - the official postmortem: August 17, 13:40 UTC, 7 hours 35 minutes, the sidecar concurrency limit, and the latent client retry bug.
2. Exponential Backoff And Jitter, AWS Architecture Blog - Marc Brooker’s canonical post: why plain backoff synchronizes clients, and the measured comparison of full, equal, and decorrelated jitter.
3. Retries make outages worse (Automattic) - a compact explainer of retry storms and metastable failure, with the feedback-loop diagram.
4. GitHub outage August 2026: the autoscaling failure and retry storm - secondary writeup with the reported traffic numbers: 7,000-9,000 RPS normal, 70,000-100,000 at the peak.