Exponential Backoff for AI API Calls: An Implementation Guide
5 min read · Updated Aug 5, 2026
Exponential backoff means waiting longer between each retry of a failed API call, roughly doubling the delay every attempt, plus a small random jitter, instead of retrying instantly or on a fixed schedule. For AI API calls specifically, where a 429 rate limit or a 503 overload error shows up the moment traffic spikes, backoff turns a wall of simultaneous retries into a staggered trickle the provider’s rate limiter can actually absorb.
Key takeaways
- Exponential backoff doubles (or multiplies by some base) the wait time after each failed attempt: roughly 1s, 2s, 4s, 8s.
- Add random jitter to the delay, or every client that failed at the same moment retries at the same moment again, recreating the exact spike that caused the failure.
- Only retry errors that are actually retryable: 429 (rate limited), 500 to 504 (server-side), and network timeouts. Never blindly retry a 400 or 401, the request is wrong, not the timing.
- Respect a
Retry-Afterheader when a provider sends one. It overrides your own backoff math with the server’s actual stated wait time. - Cap both the number of retries and the maximum delay, or a single failing call can hold a request open for minutes while a user stares at a spinner.
Why fixed retries make rate limits worse, not better
A rate limit error almost never happens to one client in isolation. Traffic spiked, and every client hitting the API at that moment got rate limited at roughly the same time. If all of them retry after a fixed one-second delay, they all hit the API again at the same moment, get rate limited again together, and repeat the exact pattern that caused the problem. This is the thundering herd problem, and it is the entire reason backoff needs to grow over time and vary between clients, not just add a delay.
The algorithm, step by step
Retry only on retryable status codes, calculate a delay that grows exponentially with the attempt number, add jitter so simultaneous failures spread out instead of clustering, defer to the provider’s own Retry-After header when it is present, and give up after a fixed number of attempts rather than retrying forever.
async function callWithBackoff(
fn,
{ maxRetries = 5, baseDelayMs = 1000, maxDelayMs = 30000 } = {},
) {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (err) {
const status = err.status ?? err.response?.status;
const retryable = status === 429 || (status >= 500 && status <= 504);
if (!retryable || attempt === maxRetries) throw err;
const retryAfterHeader = err.response?.headers?.['retry-after'];
const serverDelay = retryAfterHeader
? Number(retryAfterHeader) * 1000
: null;
const exponential = Math.min(baseDelayMs * 2 ** attempt, maxDelayMs);
const jitter = Math.random() * exponential * 0.5;
const delay = serverDelay ?? exponential + jitter;
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
}How many retries, and how long to wait between them
| Attempt | Exponential delay | With jitter applied |
|---|---|---|
| 1 (first retry) | 1s | 1.0 to 1.5s |
| 2 | 2s | 2.0 to 3.0s |
| 3 | 4s | 4.0 to 6.0s |
| 4 | 8s | 8.0 to 12.0s |
| 5 (final retry) | 16s | 16.0 to 24.0s |
The 40 minutes I spent watching a queue that would not drain
In August 2023, I had a content-summarization pipeline calling the OpenAI API with a fixed one-second retry on failure, no jitter, no growth. A traffic spike triggered rate limiting, and three separate queue workers running identical code all retried at the exact same one-second mark, over and over, in lockstep. The queue backed up for about 40 minutes and burned through roughly 12,000 wasted retry requests before I noticed the pattern in the logs. Adding jitter alone, without even touching the exponential growth, fixed it within the next deploy: the workers stopped synchronizing and the backlog cleared in under five minutes.
How many retries is too many
I think most teams implement the backoff math correctly and then undermine it by setting the retry count too high. Five retries with a 30 second cap plus jitter is plenty for almost any pipeline. I have seen teams configure ten or more retries "to be safe" and end up holding a queue worker hostage for four or five minutes on a request that was never going to succeed, usually because the underlying error was not actually transient (a malformed prompt, an expired key) and no amount of waiting was going to fix it.
AWS’s architecture blog post "Exponential Backoff and Jitter" (Marc Brooker, 2015) is still the reference most engineers cite for this pattern a decade later, and its core finding holds up: full jitter, where the delay is a random value between zero and the exponential cap rather than the exponential value plus a small jitter on top, consistently outperforms both fixed-delay and jitter-free exponential retries under real contention.
“A retry without backoff is not resilience, it is the same failure asking to happen again slightly faster.”
Frequently asked questions
Frequently asked questions
What is exponential backoff?
A retry strategy where the wait time between attempts roughly doubles after each failure, instead of retrying instantly or on a fixed schedule. It gives an overloaded or rate-limited API time to recover before the next attempt lands.
Why add jitter to exponential backoff?
Without jitter, every client that failed at the same moment retries at the same moment again, since they are all running the same deterministic delay calculation. Random jitter spreads those retries out over time so they do not recreate the same traffic spike that caused the original failure.
Which HTTP errors should trigger a retry?
429 (rate limited) and 500 to 504 (server-side errors), along with network timeouts. These typically indicate a transient condition rather than a problem with the request itself.
Should I retry a 400 or 401 error?
No. A 400 means the request itself is malformed and a 401 means authentication failed, and retrying the identical request will produce the identical error every time. Fix the request or the credentials instead of retrying.
How many retries is too many?
Beyond about five retries with a capped maximum delay, additional attempts mostly just delay the inevitable failure while holding a request or queue worker open. If five well-spaced retries have not succeeded, the underlying issue is usually not something more waiting will fix.
What does the Retry-After header do?
Some APIs, including OpenAI and Anthropic, return a Retry-After header on rate-limited responses stating how many seconds to wait before retrying. It reflects the provider’s actual rate-limit state and should override your own exponential calculation when present.