Retry & circuit breaker for AI agents in production

AI Integration

•14 de julio de 2026•6 min read•Por Daily Miranda Pardo

Your agent is live in production. At 2:14 AM, the LLM API returns a 529 overloaded. Nobody is watching. The agent tries the call, gets the error, and the entire flow breaks. The customer waiting for a two-second response gets nothing. The downstream process that depended on the agent's output stays blocked.

What does your agent do when the infrastructure fails? If the answer is "it crashes," you have a design problem, not a code problem.

Why production agents need a failure strategy

In local and staging, APIs always respond. In production, they don't. LLM APIs fail for several reasons that happen more often than the documentation admits:

Rate limits (429 Too Many Requests): too many requests in a short time window
Provider overload (529 Overloaded): happens with Anthropic, OpenAI, and everyone else
Network timeouts (ETIMEDOUT): the connection opens but the response never arrives
Transient server errors (500, 503): the provider has a momentary issue

Most of these errors are transient: retry 2 seconds later and it works. The problem is that most agents are implemented with zero retry logic. A single failure brings down the entire flow.

Two patterns solve this in production: retry with exponential backoff and circuit breaker. They're not alternatives — you use them together.

Retry with exponential backoff and jitter

The basic idea of retry is simple: if the call fails, wait a bit and try again. The mistake is in the naive implementation that retries immediately in a tight loop.

If your agent fails and retries at the exact same moment, and there are 200 agents doing the same thing because the provider is saturated, you've just multiplied the problem by 200. This is called thundering herd and turns a temporary problem into a sustained outage.

The solution: exponential backoff with jitter. Each attempt waits longer than the previous one, and a random component (jitter) is added so that retries from multiple instances don't synchronize.

interface RetryOptions {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  retryableStatuses: number[];
}

async function withRetry<T>(
  fn: () => Promise<T>,
  options: RetryOptions = {
    maxAttempts: 4,
    baseDelayMs: 1000,
    maxDelayMs: 16000,
    retryableStatuses: [429, 500, 503, 529],
  }
): Promise<T> {
  let lastError: Error;

  for (let attempt = 1; attempt <= options.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error as Error;
      const status = (error as any)?.status;

      // Don't retry permanent errors (400, 401, 404...)
      if (status && !options.retryableStatuses.includes(status)) {
        throw error;
      }

      if (attempt === options.maxAttempts) break;

      // Exponential backoff: 1s, 2s, 4s, 8s...
      const exponentialDelay = options.baseDelayMs * Math.pow(2, attempt - 1);
      // Jitter: ±30% of the calculated delay
      const jitter = exponentialDelay * 0.3 * (Math.random() * 2 - 1);
      const delay = Math.min(
        Math.round(exponentialDelay + jitter),
        options.maxDelayMs
      );

      console.warn(`[retry] attempt ${attempt} failed (${status}), waiting ${delay}ms`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }

  throw lastError!;
}

// Usage in the agent
const response = await withRetry(() =>
  client.messages.create({
    model: 'claude-opus-4-8',
    max_tokens: 2048,
    messages,
  })
);

An important detail: not every error should be retried. A 400 Bad Request or 401 Unauthorized is a permanent error. Retrying a 400 with the same parameters will never work — all you do is burn credits and add latency. Only retry transient errors.

Circuit breaker: don't make calls you know will fail

Retry handles isolated failures. Circuit breaker handles when the provider is genuinely down.

If the API has been failing for 3 minutes, there's no point in every request reaching the provider, waiting for the timeout, and coming back after the backoff. That latency accumulates for the user, and the unnecessary load on the infrastructure makes things worse.

The circuit breaker monitors failures and, when they exceed a threshold, stops sending requests for a time period. Instead of making the call, it returns an immediate error. After the timeout, it lets one probe request through. If it succeeds, it returns to normal. If it fails, it extends the block.

type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

class CircuitBreaker {
  private state: CircuitState = 'CLOSED';
  private failureCount = 0;
  private lastFailureTime = 0;

  constructor(
    private readonly failureThreshold = 5,
    private readonly recoveryTimeoutMs = 30_000
  ) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'OPEN') {
      const elapsed = Date.now() - this.lastFailureTime;
      if (elapsed < this.recoveryTimeoutMs) {
        throw new Error('Circuit breaker OPEN: service unavailable');
      }
      this.state = 'HALF_OPEN';
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess() {
    this.failureCount = 0;
    this.state = 'CLOSED';
  }

  private onFailure() {
    this.failureCount++;
    this.lastFailureTime = Date.now();
    if (this.failureCount >= this.failureThreshold) {
      this.state = 'OPEN';
      console.error(`[circuit-breaker] OPEN after ${this.failureCount} failures`);
    }
  }
}

// Combined usage: circuit breaker + retry
const breaker = new CircuitBreaker(5, 30_000);

const response = await breaker.execute(() =>
  withRetry(() => client.messages.create({ model: 'claude-opus-4-8', max_tokens: 2048, messages }))
);

The circuit breaker is instantiated once per provider (not per call) and shared across all agent instances. If it's in OPEN state, all requests to that provider fail fast without hitting the network.

Dead letter queue: failures you can't recover from immediately

When an agent processes an async task — generating a report, sending a notification, updating a record — and fails even after all retries, you need to know what happened and be able to process it later.

The solution is a dead letter queue (DLQ): before discarding the task, save it to a database table with the original payload, attempt count, and last error. A monitoring process reviews the DLQ and can manually retry, alert the team, or execute compensation logic.

This connects directly to the stateful agents pattern: the agent's state must include the failed attempt history so no critical task gets silently lost.

Graceful degradation: when you can't wait for the LLM

For synchronous flows where the user is waiting for a response, you can't always afford 15-20 seconds of backoff. In those cases, the strategy is graceful degradation: if the LLM doesn't respond within X ms, return a predefined response or route to a human agent.

This is part of how we design every AI integration for businesses: each flow has a defined behavior on failure. No agent crashes silently without something handling it.

When to build this yourself vs. when not to

These patterns aren't optional if you have agents in production with any SLA. They're the difference between an agent that works in staging and one that works at 3 AM when nobody is watching.

The basic implementation you see here takes less than a day. The problem is that most teams build the agent first and add resilience "when there's time" — which in practice means never, until there's an incident.

If you're building AI agents for your business and want to make sure they handle real infrastructure failures before going live, tell me where you are:

Let's talk about building agents that don't break in production →