Handling Rate Limits and Retries in Complex API Pipelines

Dashboard showing API rate limit metrics and retry queue

You have built a beautiful API pipeline. It processes data, transforms it, and sends it to the right destination. Then one morning, everything stops working. The API returns a 429 error — Too Many Requests. Your pipeline has hit a rate limit. This is one of the most common and frustrating problems in API development, and solving it properly makes the difference between amateur and professional pipelines.

What Are Rate Limits and Why Do They Exist?

Rate limits are caps on how many API requests you can make in a given time window. An API might allow 100 requests per minute, 1,000 per hour, or 10,000 per day. They exist to protect the API provider from being overwhelmed and to ensure fair usage across all customers. When you exceed the limit, the API returns a 429 status code and usually includes a Retry-After header telling you how long to wait.

Exponential Backoff: The Core Pattern

The most important retry pattern is exponential backoff. When a request fails, wait 1 second and try again. If it fails again, wait 2 seconds. Then 4 seconds. Then 8 seconds. Each retry doubles the wait time. Add a small random delay (called jitter) to prevent all your retries from hitting the API at exactly the same moment. Cap the maximum wait at around 60 seconds and set a maximum retry count of 3 to 5 attempts.

  • Catch the 429 response and read the Retry-After header if present
  • If no Retry-After header, calculate wait time as 2^attempt * 1000ms plus random jitter
  • Wait for the calculated duration, then retry the request
  • If the retry also fails, double the wait time and try again up to the max retry count
  • After exhausting all retries, log the failure and move it to a dead letter queue for manual review
Code editor showing exponential backoff retry implementation

Proactive Rate Limit Management

The best approach is to avoid hitting rate limits in the first place. Track your API usage with counters and throttle your requests to stay under the limit. Many APIs include rate limit headers in every response — X-RateLimit-Remaining tells you how many requests you have left, and X-RateLimit-Reset tells you when the window resets. Use these values to pace your pipeline automatically.

Queue-Based Architecture

For high-volume pipelines, use a queue to buffer requests. Instead of firing API calls directly, push them onto a queue like BullMQ, SQS, or even a simple Redis list. A worker process pulls from the queue at a controlled rate that respects the API limit. If a request fails, it goes back into the queue with a delay. This pattern naturally prevents rate limit errors and makes your pipeline resilient to API outages.

Rate Limits in n8n Workflows

n8n has built-in retry settings on every node. You can set the number of retries and the wait time between attempts. For more advanced control, use the Wait node together with a Loop to implement custom backoff logic. Combine this with the Error Trigger workflow to catch and handle failures across all your automations from a single place.