Context Summarization for AI Agents: Scale Long Chats
Your support agent is forty messages into a back-and-forth with a customer. The context exceeds 40,000 tokens. On message forty-one, the model no longer remembers the agreement reached in message twelve.
This isn't an LLM bug. It's the consequence of an architecture that doesn't scale.
Most teams building conversational agents pass the full history as a messages array. It works in development. In production, as conversations grow longer, the problems start.
Why Passing the Full History Is an Architecture Mistake
The message array grows with every turn. Every model call includes everything that came before. Three symptoms appear, sooner or later:
The agent starts "forgetting" the beginning. Models give more weight to recent context. A 40,000-token context where key facts are in the first 10% is a poorly distributed context. The model pays less attention to them.
Cost per call grows exponentially. A twenty-turn conversation can cost ten times more than the same conversation if the history is unchecked. At production volume, that becomes real money.
Context length errors hit at the worst moment. The user is in the middle of an important transaction. The API returns context_length_exceeded. The conversation dies.
The pattern that solves this has a name: progressive summarization. Nothing new in information systems, but its implementation for LLM agents has nuances worth knowing.
Three Strategies, Ranked from Worst to Best
Truncation: The Most Common Mistake
The emergency fix that lives in everyone's codebase:
function buildMessages(history: Message[], maxMessages = 20): Message[] {
return history.slice(-maxMessages);
}
It discards older messages. Simple, fast, and destroys critical information. If the customer shared their account number in message three, by message twenty-three it no longer exists for the agent. The model improvises or asks again. The customer gets frustrated.
Rolling Window: Better, But Still Insufficient
A slightly smarter variation: keep the last N messages by token budget, not by count:
function trimToTokenBudget(
messages: Message[],
maxTokens: number,
estimatedTokensPerChar = 0.25
): Message[] {
let total = 0;
const result: Message[] = [];
for (const msg of [...messages].reverse()) {
const estimated = msg.content.length * estimatedTokensPerChar;
if (total + estimated > maxTokens) break;
result.unshift(msg);
total += estimated;
}
return result;
}
Better than fixed-count truncation, but still loses information. The very first message where the customer explained their issue might be exactly what the model needs to avoid asking them to repeat it.
Progressive Summarization: The Pattern That Scales
The idea is simple: when the history exceeds a threshold, generate a summary of the oldest messages and replace them with that summary. Recent messages stay complete. The result:
- The model has access to all relevant conversation information
- Context never exceeds your defined threshold
- Cost per call becomes predictable
TypeScript Implementation
import Anthropic from '@anthropic-ai/sdk';
interface Message {
role: 'user' | 'assistant';
content: string;
}
interface ConversationState {
summary: string | null;
recentMessages: Message[];
totalTurns: number;
}
const anthropic = new Anthropic();
async function summarizeMessages(messages: Message[]): Promise<string> {
const response = await anthropic.messages.create({
model: 'claude-haiku-4-5-20251001',
max_tokens: 512,
system:
'You are a conversation compression system. Summarize the key points of these messages in 3-5 concise sentences. Preserve: decisions made, concrete data mentioned (names, numbers, dates), and the current state of the issue. Do not include greetings or generic phrases.',
messages: [
{
role: 'user',
content: `Summarize this conversation:\n\n${messages
.map((m) => `${m.role === 'user' ? 'Customer' : 'Agent'}: ${m.content}`)
.join('\n')}`,
},
],
});
return response.content[0].type === 'text' ? response.content[0].text : '';
}
class ConversationManager {
private state: ConversationState = {
summary: null,
recentMessages: [],
totalTurns: 0,
};
private readonly maxRecentMessages: number;
constructor(maxRecentMessages = 12) {
this.maxRecentMessages = maxRecentMessages;
}
async addTurn(userMessage: string, agentResponse: string): Promise<void> {
this.state.recentMessages.push(
{ role: 'user', content: userMessage },
{ role: 'assistant', content: agentResponse }
);
this.state.totalTurns++;
if (this.state.recentMessages.length > this.maxRecentMessages) {
await this.compress();
}
}
private async compress(): Promise<void> {
const toSummarize = this.state.recentMessages.slice(
0,
Math.floor(this.maxRecentMessages / 2)
);
const newSummary = await summarizeMessages(toSummarize);
const combinedSummary = this.state.summary
? `Previous context: ${this.state.summary}\n\nUpdate: ${newSummary}`
: newSummary;
this.state.summary = combinedSummary;
this.state.recentMessages = this.state.recentMessages.slice(
Math.floor(this.maxRecentMessages / 2)
);
}
buildMessagesForLLM(): Message[] {
const messages: Message[] = [];
if (this.state.summary) {
messages.push({
role: 'user',
content: `[Conversation summary so far]\n${this.state.summary}`,
});
messages.push({
role: 'assistant',
content: 'Understood. Continuing from that point.',
});
}
return [...messages, ...this.state.recentMessages];
}
getStats() {
return {
turns: this.state.totalTurns,
recentMessages: this.state.recentMessages.length,
hasSummary: !!this.state.summary,
summaryLength: this.state.summary?.length ?? 0,
};
}
}
Integration in a Next.js Route
// app/api/chat/route.ts
import { ConversationManager } from '@/lib/conversation-manager';
const sessions = new Map<string, ConversationManager>();
export async function POST(req: Request) {
const { sessionId, message } = await req.json();
let manager = sessions.get(sessionId);
if (!manager) {
manager = new ConversationManager(12);
sessions.set(sessionId, manager);
}
const messages = manager.buildMessagesForLLM();
const response = await anthropic.messages.create({
model: 'claude-sonnet-4-6',
max_tokens: 1024,
system: 'You are the company support assistant...',
messages: [...messages, { role: 'user', content: message }],
});
const agentReply =
response.content[0].type === 'text' ? response.content[0].text : '';
await manager.addTurn(message, agentReply);
return Response.json({
reply: agentReply,
stats: manager.getStats(),
});
}
One important production note: the ConversationManager is instantiated per session and state must persist between requests. With Vercel's serverless architecture, an in-memory Map doesn't survive between invocations — you need Redis, Supabase, or any external store.
What Changes in Production
With this pattern in production, in AI support and automation agents we build at DAILYMP, the results are consistent:
- 80–90% token reduction per call in conversations beyond twenty turns
- Predictable cost per conversation regardless of length
- Zero context length errors in production
- Maintained coherence across fifty, one hundred, or more turns
The key is the model you use for summarization. Claude Haiku is fast and good enough for history compression — there's no reason to use an expensive model for this auxiliary task. The cost of generating the summary is marginal compared to what you save on every main call.
Something that consistently surprises teams: the summary doesn't have to be perfect. You don't need the compression model to deeply understand your domain. You need it to preserve dates, names, decisions, and the state of the problem. Any modern model does this well with a focused compression system prompt.
Common Mistakes When Implementing This Pattern
Summarizing too early. If the threshold is low (four or six messages), the compression cost starts competing with the savings. In practice, ten to sixteen recent messages before compressing works well for most support use cases.
Not versioning the summary. If the agent's system prompt changes, summaries generated with the previous version may have a different format or focus. Store the compression prompt version alongside the summary. Invalidate stale summaries.
Losing system context. The summary covers the message history, not the agent's business context (rules, catalog, specific customer). That context still lives in the main system prompt, not in the summary.
For more complex architectures — agents with tools, multitenancy, or sessions that need to survive for days — the pattern can be extended with a key facts extraction store complementary to the summary. This is what we implement in our AI integration services when the agent needs to remember structured user-specific information.
When You Need This
If your agent handles conversations of more than ten turns in production and you don't have context summarization yet, you're already overpaying on every call and your users are already noticing inconsistencies.
No major refactor needed. The ConversationManager above can be added to an existing agent in a few hours — and the return is immediate in cost and conversation quality.
Want to implement it in your agent? Reach out on WhatsApp and let's review it together.