AI Agent Concurrency: Taming 429s, Queues & Semaphores

Your pipeline processes invoices. In staging it works fine: ten agent calls, all respond, data lands in Supabase. You deploy for a real client. The first month-end close arrives: 200 invoices in the queue. You trigger the process. Two minutes later, 70% of your calls return 429 Too Many Requests. The agent isn't broken. Your concurrency architecture is.

This failure shows up in almost every project that scales from "works locally" to "works under real load". And the root cause is always the same: firing every LLM call simultaneously, with no flow control.

The Mistake Everyone Makes First

The naive approach is tempting because it looks efficient:

// ❌ This blows up in production with more than 20 simultaneous tasks
const results = await Promise.all(
  invoices.map(invoice => callAgent(invoice))
);

Locally, with 10 test invoices, it works perfectly. In production, with 200 real invoices arriving at once, you're sending 200 simultaneous requests to the LLM API. The first 20–30 get in. The rest receive 429. Those that fail retry at the same time. The spike repeats. The system enters a retry loop that makes things worse.

This is not a model problem. It's not a business logic problem. It's a concurrency flow problem.

Anthropic's rate limits operate on two dimensions: requests per minute (RPM) and tokens per minute (TPM). Promise.all violates both simultaneously.

Pattern 1: Concurrency Semaphore

A semaphore limits how many operations run in parallel at any given moment. Those that exceed the limit wait in a queue. When one completes, it releases its slot and the next one enters.

class Semaphore {
  private queue: Array<() => void> = [];
  private running = 0;

  constructor(private readonly limit: number) {}

  async acquire(): Promise<void> {
    if (this.running < this.limit) {
      this.running++;
      return;
    }
    await new Promise<void>(resolve => this.queue.push(resolve));
    this.running++;
  }

  release(): void {
    this.running--;
    const next = this.queue.shift();
    if (next) next();
  }
}

async function withSemaphore<T>(
  tasks: Array<() => Promise<T>>,
  limit: number
): Promise<T[]> {
  const sem = new Semaphore(limit);
  return Promise.all(
    tasks.map(async task => {
      await sem.acquire();
      try {
        return await task();
      } finally {
        sem.release();
      }
    })
  );
}

// ✅ Maximum 5 agent calls running in parallel
const results = await withSemaphore(
  invoices.map(inv => () => callAgent(inv)),
  5
);

With limit: 5, instead of firing 200 calls at once, you keep exactly 5 active at all times. When one finishes, the next enters. Total time is longer, but the pipeline doesn't collapse.

The right limit depends on your API tier. On Anthropic's standard tier, 3–8 parallel calls is typically safe. Higher tiers can sustain 20–30.

Pattern 2: Retry with Exponential Backoff and Jitter

The semaphore fixes the structural problem, but transient 429s still appear: traffic spikes, moments of API pressure, windows where your app isn't the only consumer. You need a retry layer that doesn't generate a retry storm.

The classic mistake: wait a fixed second and retry. If 10 calls fail simultaneously and all wait exactly 1 second, they hit the API again at the same time. The 429 repeats.

The solution: exponential backoff with jitter (random noise).

async function withRetry<T>(
  fn: () => Promise<T>,
  maxAttempts = 4,
  baseDelayMs = 1000
): Promise<T> {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err: any) {
      const isRateLimit = err?.status === 429 || err?.error?.type === 'rate_limit_error';
      if (!isRateLimit || attempt === maxAttempts - 1) throw err;

      // Exponential backoff: 1s, 2s, 4s, 8s... plus random jitter
      const jitter = Math.random() * 500;
      const delay = baseDelayMs * Math.pow(2, attempt) + jitter;
      await new Promise(r => setTimeout(r, delay));
    }
  }
  throw new Error('Max retries reached');
}

The jitter ensures each retry has a slightly different delay from the others. Calls that failed together no longer retry together. The retry storm disappears.

Combine both patterns for full control:

const results = await withSemaphore(
  invoices.map(inv => () => withRetry(() => callAgent(inv))),
  5
);

Pattern 3: Priority Queue

In production, not all tasks are equal. A user waiting for a real-time response shouldn't compete for the same slots as a batch job processing 500 documents that could run in the background.

A priority queue solves exactly this: urgent tasks get processed first, regardless of arrival order.

type Priority = 'high' | 'normal' | 'low';

interface Task<T> {
  fn: () => Promise<T>;
  priority: Priority;
  resolve: (value: T) => void;
  reject: (err: unknown) => void;
}

class PriorityQueue<T> {
  private queues: Record<Priority, Task<T>[]> = {
    high: [],
    normal: [],
    low: []
  };

  enqueue(task: Task<T>): void {
    this.queues[task.priority].push(task);
  }

  dequeue(): Task<T> | undefined {
    return (
      this.queues.high.shift() ??
      this.queues.normal.shift() ??
      this.queues.low.shift()
    );
  }
}

A real-time user request gets priority: 'high'. The nightly batch process uses priority: 'low'. The agent always serves urgent work first.

The Production Failures These Patterns Prevent

The cascading 429: without a semaphore, a traffic spike sends 100 simultaneous calls, all fail, all retry at once, the spike repeats in a loop. The system becomes unresponsive for minutes.

False agent failures: when a 429 arrives as a generic error without proper handling, it looks like the agent's logic is broken. Hours of debugging to discover the cause was rate limiting, not the model.

Batch blocking real users: without priority, the process reconciling last month's 200 invoices occupies all slots while a user waits for a real-time agent response.

The three patterns work as layers: the semaphore controls volume, the retry with jitter absorbs transient spikes, and the priority queue ensures urgent work always goes first.

If you're building AI agent integration systems or automation pipelines that need to scale, these three patterns are the difference between a pipeline that works in demo and one that holds up in real production under variable load.

---

Got an agent pipeline that fails under load or traffic spikes? Tell me the scenario and I'll tell you exactly what's happening.

Let's talk about your project on WhatsApp →