Mock the LLM in Your Tests: 500 Cases, Zero API Credits
There's a strange line on your Anthropic bill this month: $47 in CI calls. Not the production agent. The tests.
You have 50 tests hitting the live Claude API. Each run takes 2 minutes and burns credits. When GitHub Actions launches three parallel runs because two PRs opened at the same time, the bill multiplies. And to top it off, one of those tests fails randomly because the LLM returned the JSON slightly differently this time.
This is the core problem with testing AI agents against the live API. It's not just about money — tests that depend on a live LLM are not reliable as a regression suite.
The common confusion: what to mock and what not to
Agent testing architecture has three layers with distinct purposes, and mixing them up is the source of all the trouble.
Unit tests (mocked LLM): validate agent logic — routing, error handling, tool call order, state transitions. No live API calls. 500 tests in 1 second, $0.
Integration tests (real LLM, mocked tools): validate that your prompts actually work as intended. Here it makes sense to call the LLM, but use the dependency injection pattern to control external tools. 20–30 tests, run on staging, minimal cost.
Evaluations (LLM-as-judge): validate output quality. These do need a real LLM evaluating another LLM. We covered this in the prompt regression CI article.
The most common mistake is merging layer 1 with layer 3: using the live API to test logic that could be verified with a deterministic mock.
How to mock the Anthropic client in Vitest
The @anthropic-ai/sdk client is an ordinary class. With vi.mock you replace it with a controlled version:
// tests/helpers/mock-llm.ts
import { vi } from 'vitest'
import type Anthropic from '@anthropic-ai/sdk'
vi.mock('@anthropic-ai/sdk', () => ({
default: vi.fn().mockImplementation(() => ({
messages: {
create: vi.fn(),
},
})),
}))
// Factory: simple text response
export function mockTextResponse(text: string): Anthropic.Message {
return {
id: 'mock-id',
type: 'message',
role: 'assistant',
content: [{ type: 'text', text }],
model: 'claude-sonnet-4-5',
stop_reason: 'end_turn',
usage: { input_tokens: 100, output_tokens: 50 },
}
}
// Factory: tool use response
export function mockToolUse(
toolName: string,
input: Record<string, unknown>
): Anthropic.Message {
return {
id: 'mock-id',
type: 'message',
role: 'assistant',
content: [{ type: 'tool_use', id: 'mock-tool-id', name: toolName, input }],
model: 'claude-sonnet-4-5',
stop_reason: 'tool_use',
usage: { input_tokens: 150, output_tokens: 80 },
}
}
With these two factories you can simulate any response sequence the LLM would return.
Testing routing logic
The most valuable test you can write with a mock: verify the agent calls tools in the correct order.
// tests/billing-agent.test.ts
import { describe, it, expect, beforeEach } from 'vitest'
import Anthropic from '@anthropic-ai/sdk'
import { runBillingAgent } from '../agents/billing-agent'
import { mockTextResponse, mockToolUse } from './helpers/mock-llm'
import { createMockTools } from './helpers/mock-tools'
describe('BillingAgent — routing', () => {
let mockCreate: ReturnType<typeof vi.fn>
let mockTools: ReturnType<typeof createMockTools>
beforeEach(() => {
const client = new Anthropic()
mockCreate = client.messages.create as ReturnType<typeof vi.fn>
mockTools = createMockTools()
})
it('fetches customer before creating invoice', async () => {
// LLM decides: search first, then create
mockCreate
.mockResolvedValueOnce(mockToolUse('getCustomer', { id: 'c123' }))
.mockResolvedValueOnce(mockToolUse('createInvoice', { amount: 500 }))
.mockResolvedValueOnce(mockTextResponse('Invoice created successfully.'))
const result = await runBillingAgent(
'Create a $500 invoice for customer c123',
mockTools
)
expect(result.toolCallOrder).toEqual(['getCustomer', 'createInvoice'])
expect(mockCreate).toHaveBeenCalledTimes(3)
})
})
This test costs nothing. It runs in 2ms. And it verifies something you can't check with output-only tests: the order in which the agent makes decisions.
Testing error paths nobody tests
The most valuable part of mocking is being able to trigger situations that are rare in production but devastating when they happen.
it('falls back gracefully when LLM returns invalid JSON', async () => {
mockCreate.mockResolvedValueOnce(
mockTextResponse('{ "action": "create_invoice", invalid json here }')
)
const result = await runBillingAgent('Create invoice', mockTools)
expect(result.status).toBe('fallback')
expect(mockTools.email.send).not.toHaveBeenCalled() // nothing was sent
})
it('retries when LLM returns stop_reason: max_tokens', async () => {
mockCreate
.mockResolvedValueOnce({ ...mockTextResponse('...'), stop_reason: 'max_tokens' })
.mockResolvedValueOnce(mockTextResponse('Complete response.'))
const result = await runBillingAgent('Generate a long report', mockTools)
expect(result.status).toBe('success')
expect(mockCreate).toHaveBeenCalledTimes(2) // retried
})
These error paths are nearly impossible to test reliably against the live API. With mocks, they're trivial and run in milliseconds.
The number that justifies everything
With live API: 50 tests, 2-minute runs, $12 per CI run. With 20 PRs per week, that's $240 weekly just for tests — not counting the slowdown in developer feedback loops.
With deterministic mocks: 500 tests covering triple the cases, 1.2 seconds execution time, $0.
Integration tests with the real LLM are still necessary — but that's 20 tests to validate that prompts work, not 500 routing-logic tests that can run without an API.
The right architecture is not a choice between speed and confidence. It's using each tool where it makes sense: mocks for logic, real LLM for prompts, LLM-as-judge for quality.
The cost of getting here late
Most projects we see at DAILYMP arrive at this testing infrastructure after they already have a production agent with testing technical debt. Adding mocking after the fact to an agent with hardcoded dependencies is expensive — you have to refactor before you can test.
The right architecture is decided upfront: dependency injection for tools, mock factories for the LLM, and all three test levels from sprint one.
Building it right from scratch takes a day. Rebuilding it in an existing agent can take a week.
Building an AI agent and don't have this test infrastructure? In 30 minutes I'll walk you through exactly what you need for a CI-ready test suite that doesn't burn API credits.