LLM Provider Migration: Switch Vendors Without Breaking Production
You've been on one provider for twelve months. You know the quirks, the stack is dialed in, the agent works. Then something changes: costs increase, a better model ships elsewhere, your team needs features your current provider doesn't offer. You decide to migrate.
You swap the client, adjust the parameters, deploy. And the agent that ran cleanly for a year starts misbehaving — wrong tool selections, silent failures in cases that never broke before.
The issue isn't the quality of the new provider. It's that every LLM provider speaks a different dialect, and nobody warned you about the production-level details that actually matter.
Why LLM APIs Are Not Interchangeable
On paper it looks simple: both providers receive messages and return responses. In production, the differences bite harder than you'd expect:
Tool calling format:
- OpenAI:
finish_reason: "tool_calls", response inmessage.tool_calls[] - Anthropic:
stop_reason: "tool_use", response incontent[]astool_useblocks
Token parameter:
- OpenAI:
max_completion_tokens(the oldmax_tokensis deprecated) - Anthropic:
max_tokens(required — no default, omitting it throws a 400)
Streaming event format:
- OpenAI:
ChatCompletionChunkevents with cumulative delta - Anthropic:
content_block_deltaevents with explicit block index
Tool result messages:
- OpenAI: message with
role: "tool"and atool_call_id - Anthropic:
tool_resultcontent block inside a user message
Any of these differences, if you miss it, produces failures that throw no exceptions but silently corrupt your agent's logic.
The Pattern: Abstraction Layer + Gradual Traffic Migration
The solution used in DAILYMP's AI agent integration projects has two parts: a common interface that normalizes differences, and a traffic router that moves users incrementally to the new provider.
Step 1: Define a provider-agnostic interface
// lib/llm/types.ts
export interface LLMRequest {
messages: Message[];
tools?: ToolDefinition[];
maxTokens?: number;
temperature?: number;
systemPrompt?: string;
}
export interface LLMResponse {
content: string;
toolCalls?: ToolCall[];
stopReason: 'end_turn' | 'tool_use' | 'max_tokens';
usage: { inputTokens: number; outputTokens: number };
}
export interface LLMClient {
complete(req: LLMRequest): Promise<LLMResponse>;
stream(req: LLMRequest): AsyncIterable<LLMChunk>;
}
Step 2: Implement one adapter per provider
Each adapter translates your common interface into the provider's native format. Your agent code never imports a provider SDK directly — it only depends on LLMClient.
// lib/llm/anthropic-client.ts
export class AnthropicClient implements LLMClient {
private client = new Anthropic();
async complete(req: LLMRequest): Promise<LLMResponse> {
const res = await this.client.messages.create({
model: 'claude-sonnet-4-6',
max_tokens: req.maxTokens ?? 4096,
system: req.systemPrompt,
messages: req.messages.map(adaptMessageToAnthropic),
tools: req.tools?.map(adaptToolToAnthropic),
});
return {
content: extractTextContent(res.content),
toolCalls: extractToolCalls(res.content),
stopReason: mapStopReason(res.stop_reason),
usage: { inputTokens: res.usage.input_tokens, outputTokens: res.usage.output_tokens },
};
}
}
The OpenAI adapter follows the same structure, speaking OpenAI's dialect. Your agent changes zero lines when you swap providers.
Step 3: Traffic router with gradual rollout
You don't flip 100% of traffic at once. Start at 5%, watch evals and metrics, then step up:
// lib/llm/router.ts
export function getLLMClient(userId: string): LLMClient {
const rolloutPercent = getFeatureFlag('llm_provider_b_rollout'); // 5 → 20 → 50 → 100
const bucket = hashUserId(userId) % 100;
if (bucket < rolloutPercent) {
return new AnthropicClient();
}
return new OpenAIClient();
}
// In your agent handler:
const llm = getLLMClient(session.userId);
const response = await llm.complete(request);
With a deterministic hashUserId, the same user always hits the same provider throughout the rollout. This matters for multi-turn conversation coherence and for avoiding inconsistent behavior within the same user session.
Three Gotchas Nobody Warns You About
1. Tool names must be unique and have no spaces
Anthropic validates tool names more strictly than OpenAI. If you have tools named like "get customer data" (with a space), Anthropic returns a 400. Audit your tool definitions before starting.
2. Anthropic's system prompt is a separate field
With OpenAI, the system prompt is a message with role: "system" inside the messages array. With Anthropic, it's a top-level system field. Your adapter must extract it correctly, or the agent silently ignores its own instructions.
3. User and assistant messages must alternate
Anthropic strictly validates that messages alternate between user and assistant. Two consecutive user messages — which OpenAI permits — will fail immediately. Normalize conversation history in your adapter before passing it through.
When to Gate on Evals
In a gradual migration, your eval suite is the entry gate to each rollout phase. Before stepping from 5% to 20%, you run evals against the 5% cohort's traffic and confirm the score hasn't dropped. If it drops, you roll back to 0% in seconds.
For a well-instrumented production agent, a full provider migration using this approach typically takes two to three weeks. It's a slow, methodical, uneventful process. Exactly as it should be.
Why Getting the Architecture Right Upfront Matters
If you already have a production agent without an abstraction layer, you're carrying technical debt that will collect at the worst possible time. The next time a provider raises prices, deprecates an API version, or a better model ships elsewhere, you'll be rewriting the core of your system instead of changing one line of config.
In the projects we build at DAILYMP, this pattern ships on day one. Not as an exotic architectural decision — as basic hygiene for a system that will run in production for years.
If you have an agent running on a single provider and are thinking about migration, or want to check whether your current architecture can absorb a switch without cost, let's talk.