Circuit Breaker for AI Agents: Stop Retrying at Scale

Your agent calls the LLM. It fails. Retries. Fails again. Four retries with exponential backoff: 1s, 2s, 4s, 8s. Fifteen seconds of waiting per request. While 40 concurrent users hit the same wall.

The LLM has been down for two minutes. No retry is going to fix that. But your system keeps trying — 40 times a second — piling load on a service that isn't responding.

Retry protects against transient failures. Circuit breakers protect against sustained ones. Both are necessary in production. Most teams have the first. Very few have the second.

The Cascade Nobody Sees in the Logs

The LLM outage lasts 90 seconds. Your system's impact lasts 10 minutes.

Without a circuit breaker, every incoming request spawns its own retry chain. Timeouts pile up. The connection pool saturates. Response times degrade across your entire application — not just the LLM calls. A 90-second provider outage becomes a 10-minute incident because every request keeps retrying until exhausted.

The circuit breaker breaks that cycle. When it detects sustained failure, it stops trying. Requests during the outage fail fast — in milliseconds, not 15 seconds — leaving room for a fallback response without overwhelming the system.

This is exactly what the fallback chain pattern assumes when it references circuitBreaker.isHealthy(). This article builds that component.

Three States, One Rule

The circuit breaker has three states with deterministic transitions:

CLOSED (normal operation): requests pass through to the LLM. Failures are counted within a sliding time window. When the failure count exceeds the threshold, it transitions to OPEN.

OPEN (stopped): requests fail immediately without reaching the LLM. The system responds in milliseconds with a fallback. After a configurable timeout (typically 30 seconds), it transitions to HALF_OPEN to probe recovery.

HALF_OPEN (probing): one test request goes through. Success closes the circuit and returns to normal operation. Any failure reopens it.

The central rule: in OPEN state, fail fast and serve an alternative. Paired with a proper fallback chain in your AI agent integration, users don't notice the provider outage.

Full TypeScript Implementation

type CircuitState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

interface CircuitBreakerConfig {
  failureThreshold: number;
  monitoringWindow: number;
  resetTimeout: number;
  halfOpenSuccessThreshold: number;
}

class CircuitBreaker {
  private state: CircuitState = 'CLOSED';
  private failureTimestamps: number[] = [];
  private halfOpenSuccesses = 0;
  private openedAt: number | null = null;

  constructor(
    readonly name: string,
    private readonly config: CircuitBreakerConfig
  ) {}

  isHealthy(): boolean {
    this.maybeTransitionToHalfOpen();
    return this.state !== 'OPEN';
  }

  recordFailure(): void {
    const now = Date.now();
    this.failureTimestamps = this.failureTimestamps.filter(
      ts => now - ts < this.config.monitoringWindow
    );
    this.failureTimestamps.push(now);

    if (this.state === 'HALF_OPEN') {
      this.open();
    } else if (this.failureTimestamps.length >= this.config.failureThreshold) {
      this.open();
    }
  }

  recordSuccess(): void {
    if (this.state === 'HALF_OPEN') {
      this.halfOpenSuccesses++;
      if (this.halfOpenSuccesses >= this.config.halfOpenSuccessThreshold) {
        this.close();
      }
    }
  }

  private maybeTransitionToHalfOpen(): void {
    if (
      this.state === 'OPEN' &&
      this.openedAt !== null &&
      Date.now() - this.openedAt >= this.config.resetTimeout
    ) {
      this.state = 'HALF_OPEN';
      this.halfOpenSuccesses = 0;
    }
  }

  private open(): void {
    this.state = 'OPEN';
    this.openedAt = Date.now();
    console.warn(`[CircuitBreaker:${this.name}] OPEN — ${this.failureTimestamps.length} failures in window`);
  }

  private close(): void {
    const recoveryTime = this.openedAt ? Date.now() - this.openedAt : 0;
    this.state = 'CLOSED';
    this.failureTimestamps = [];
    this.halfOpenSuccesses = 0;
    this.openedAt = null;
    console.info(`[CircuitBreaker:${this.name}] CLOSED — recovered in ${recoveryTime}ms`);
  }
}

Recommended starting parameters: failureThreshold: 5, monitoringWindow: 60_000, resetTimeout: 30_000, halfOpenSuccessThreshold: 2.

Wiring It to the Anthropic SDK in Next.js

// lib/circuit-breakers.ts — module-level singleton
export const anthropicBreaker = new CircuitBreaker('anthropic', {
  failureThreshold: 5,
  monitoringWindow: 60_000,
  resetTimeout: 30_000,
  halfOpenSuccessThreshold: 2,
});

// app/api/agent/route.ts
import Anthropic from '@anthropic-ai/sdk';
import { anthropicBreaker } from '@/lib/circuit-breakers';

const client = new Anthropic();

export async function callAgent(prompt: string): Promise<string> {
  if (!anthropicBreaker.isHealthy()) {
    return getFallbackResponse(prompt);
  }

  try {
    const response = await client.messages.create({
      model: 'claude-sonnet-4-6',
      max_tokens: 1024,
      messages: [{ role: 'user', content: prompt }],
    });

    anthropicBreaker.recordSuccess();
    const content = response.content[0];
    return content.type === 'text' ? content.text : '';

  } catch (error) {
    anthropicBreaker.recordFailure();
    throw error;
  }
}

The Detail That Breaks Everything

The circuit breaker must be a module-level singleton. Instantiate it per-request and each request gets its own failure counter — the circuit never opens.

// ❌ WRONG — each request has its own counter
export async function POST(request: Request) {
  const breaker = new CircuitBreaker('anthropic', config); // never opens
}

// ✅ CORRECT — import the shared singleton
import { anthropicBreaker } from '@/lib/circuit-breakers';

On Vercel Edge with multiple concurrent instances, the circuit state needs an external store (Redis or Vercel KV) to be shared across instances. On a single-process server or Docker container, the module singleton is sufficient.

What to Monitor

Two metrics to start: circuit open duration and fallback response rate.

State transitions are already logged in the implementation above. For a small team, an alert when the circuit has been open for more than 60 seconds covers 90% of relevant cases. The openedAt timestamp gives you the provider incident duration directly, without additional instrumentation.

As you scale, add the percentage of requests being served from fallback. If it exceeds 5% sustained, something beyond a simple provider failure is happening.

Wrapping Up

The circuit breaker doesn't prevent LLM failures. It isolates them. Your system stops amplifying them, stops saturating under load, and can serve alternatives while waiting for recovery.

The full implementation fits in under 60 lines of TypeScript. The cost of not having it is a 90-second provider outage that stretches into a 10-minute incident — every time.

Does your production agent have a circuit breaker? If you're not sure, it probably doesn't. Let's review your system's architecture together →