A/B Testing AI Agents: Compare Prompts in Production
You've spent two weeks refining your agent's system prompt. You adjust the tone, add examples, reorder instructions. You deploy. Did it improve? You feel like it did. But that's not a metric.
The problem isn't that you don't care about quality — it's that you have no system for measuring it in live traffic. Evals run against fifty test cases. Production has thousands of inputs you've never seen. And when something silently degrades, you find out weeks later through a frustrated user.
The real cost of not knowing whether you improved
Picture this cycle: change the prompt, deploy, observe for two weeks, "seems better," move on. Three months later your agent has accumulated changes with no systematic record and no measured interaction between them.
What actually happens in most teams:
- One change improves date extraction and breaks tone in negative responses
- Upgrading the model from
claude-sonnet-4toclaude-sonnet-5reduces latency but shifts behavior in edge cases - The prompt that worked in dev fails in production because real inputs carry noise your tests never had
Without systematic measurement, you're accumulating invisible quality debt.
The shadow deployment pattern for AI agents
The solution is running two versions in parallel: the active version serves the user, the candidate version processes the same input in the background. Both responses are logged. You measure which one wins.
type AgentVersion = 'v1' | 'v2';
interface AgentConfig {
systemPrompt: string;
model: string;
tools: Tool[];
}
const configs: Record<AgentVersion, AgentConfig> = {
v1: {
systemPrompt: 'You are a contracts assistant...',
model: 'claude-sonnet-4-6',
tools: [summarizeContract, extractDates],
},
v2: {
systemPrompt: 'You are a senior attorney specializing in commercial contracts...',
model: 'claude-sonnet-5',
tools: [summarizeContract, extractDates, flagRisks],
},
};
async function runWithShadow(input: string): Promise<AgentResponse> {
const [v1Result, v2Result] = await Promise.allSettled([
runAgent(input, configs.v1),
runAgent(input, configs.v2),
]);
// We serve v1 to the user
const activeResult = v1Result.status === 'fulfilled' ? v1Result.value : fallback;
// Log both for async analysis
await logShadowRun({
runId: crypto.randomUUID(),
input,
v1: v1Result.status === 'fulfilled' ? v1Result.value : null,
v2: v2Result.status === 'fulfilled' ? v2Result.value : null,
timestamp: new Date().toISOString(),
});
return activeResult;
}
Key implementation details
Promise.allSettled instead of Promise.all ensures that if v2 fails, v1 keeps serving the user without interruption. The shadow can never degrade the active experience.
Results are persisted asynchronously — if the database write is slow, it doesn't block the response to the user.
Measuring which version wins: automatic scoring
Logging responses is half the work. The other half is scoring which one is better. Three objective metrics you can compute without human intervention:
interface ShadowScore {
latencyMs: number;
tokenCount: number;
judgeScore: number; // 0-100, evaluated by another LLM
}
async function scoreShadowRun(run: ShadowRun): Promise<void> {
const [v1Score, v2Score] = await Promise.all([
computeScore(run.v1),
computeScore(run.v2),
]);
const winner = determineWinner(v1Score, v2Score);
await db.shadowResults.create({
data: {
runId: run.runId,
v1Score,
v2Score,
winner,
},
});
}
function determineWinner(
v1: ShadowScore,
v2: ShadowScore
): AgentVersion | 'tie' {
// Configurable weights based on business priority
const v1Total =
(100 - v1.latencyMs / 50) * 0.2 +
(100 - v1.tokenCount / 20) * 0.3 +
v1.judgeScore * 0.5;
const v2Total =
(100 - v2.latencyMs / 50) * 0.2 +
(100 - v2.tokenCount / 20) * 0.3 +
v2.judgeScore * 0.5;
if (Math.abs(v1Total - v2Total) < 3) return 'tie';
return v1Total > v2Total ? 'v1' : 'v2';
}
The judgeScore is computed by another LLM with a structured evaluation prompt: is the response correct, complete, and hallucination-free? This LLM-as-judge can run in batch using the Anthropic Batches API to cut costs by 50%.
When to promote v2 to production
The promotion decision should not be manual or instant. A reasonable threshold:
async function shouldPromoteV2(since: Date): Promise<boolean> {
const results = await db.shadowResults.findMany({
where: { createdAt: { gte: since } },
});
const total = results.length;
if (total < 200) return false; // statistically insufficient sample
const v2Wins = results.filter(r => r.winner === 'v2').length;
const winRate = v2Wins / total;
const v2AvgJudge =
results.reduce((sum, r) => sum + r.v2Score.judgeScore, 0) / total;
return winRate > 0.55 && v2AvgJudge > 80;
}
With 200 runs and a win rate above 55%, you have enough signal to promote with confidence. Below that and statistical noise can fool you.
Once you decide to promote, you flip configs.active = 'v2' and the shadow starts comparing v2 against the next candidate, v3.
Integrating with your existing observability stack
This pattern plugs directly into the tracing infrastructure you already have if you follow AI agent observability practices. Every shadowRun carries the same traceId as the user request, so you can cross-reference A/B test performance with business logs.
The extra cost is concrete: you're invoking the LLM twice per request. With model routing — v2 using a cheaper model during the shadow phase — the overhead can be reduced to 20-30%.
What evals don't give you but production A/B does
CI evals are essential: they catch when a change breaks something known. Production shadow testing does something different: it tells you whether the change is better for the actual inputs arriving now, with their variety and noise.
They're complementary. Evals protect against regressions. Shadow testing gives you the improvement signal.
If you're building a production agent and want to implement this pattern for your specific architecture, DAILYMP covers the full cycle: from testing strategy design to deployment and quality metrics monitoring in production.
Got an agent running and want to add this continuous validation system? Reach out directly.