Vision AI Agents: PDFs and Images in Production
Your text agent has been running in production for weeks. It works. Then the client message arrives: "We need the agent to process our scanned invoices. About 200 per month — some PDFs, some photos from a phone."
At that point, everything you built for text needs a rethink. Not because it's hard, but because multimodal pipeline failures are different — they arrive later, quieter, and more expensively than text-pipeline bugs.
What Modern LLMs Can Actually "See" (and What They Can't)
Claude, GPT-4V, and Gemini all accept images through their APIs. That doesn't mean they "read" any document the way a human does.
What works reliably in production:
- Structured field extraction from documents with consistent layouts (invoices, contracts, forms)
- Printed text at adequate resolution (≥ 150 DPI)
- Simple tables and basic spatial relationships
- Document classification by type
What fails or degrades in production:
- Low-resolution scans (photocopy of a photocopy)
- Dense handwriting or informal cursive
- Tables with more than 40 rows and no clear structure
- Multi-page PDFs treated as a single image
This isn't a temporary limitation. It's a structural characteristic of how these models work. Your preprocessing pipeline is worth as much as your prompt.
The First Mistake: Sending the PDF Directly
// ❌ The antipattern that works in demo and breaks in production
const fileBuffer = await fs.readFile('invoice.pdf');
const base64 = fileBuffer.toString('base64');
const response = await anthropic.messages.create({
model: 'claude-opus-5',
messages: [{
role: 'user',
content: [{
type: 'image',
source: { type: 'base64', media_type: 'application/pdf', data: base64 }
}, {
type: 'text',
text: 'Extract invoice number, date, subtotal, VAT and total amount.'
}]
}]
});
This breaks in production for three concrete reasons:
- File size: The API has a 5 MB limit per image. A 10-page PDF with high-quality scans can triple that.
- Multi-page: The API doesn't render intermediate pages. Depending on the model, it only processes the first page or returns an error.
- Hidden cost: A 5 MB PDF as base64 consumes ~28,000 image tokens. At 200 invoices per month, your API bill spikes before anyone notices.
The Right Pipeline: PDF → Pages → Optimised Images
import { PDFDocument } from 'pdf-lib';
import sharp from 'sharp';
async function preprocessInvoice(buffer: Buffer): Promise<string[]> {
const pdf = await PDFDocument.load(buffer);
const pageCount = pdf.getPageCount();
const base64Pages: string[] = [];
// Process only the relevant pages (invoices are usually on the first 2)
const pagesToProcess = Math.min(pageCount, 2);
for (let i = 0; i < pagesToProcess; i++) {
const singlePage = await PDFDocument.create();
const [page] = await singlePage.copyPages(pdf, [i]);
singlePage.addPage(page);
const pdfBytes = await singlePage.save();
// density: 200 = readability without inflating tokens
const imageBuffer = await sharp(pdfBytes, { density: 200 })
.png()
.resize({ width: 1200, withoutEnlargement: true })
.toBuffer();
base64Pages.push(imageBuffer.toString('base64'));
}
return base64Pages;
}
density: 200 is the sweet spot for invoices: enough resolution for small text, without bloating the payload. With resize({ width: 1200 }) you keep readability while reducing token count by around 60% compared to sending the raw PDF.
How Many Pages to Process
Define rules per document type — not a single generic value for everything:
- Invoices: maximum 2 pages
- Contracts: page 1 + signature pages
- Delivery notes: page 1 only
Processing all 20 pages of a contract to read a supplier VAT number on the first page is burning tokens for nothing. A cheap classifier (small/fast model class) decides how many pages you need before the expensive vision call.
Prompt Engineering for Vision: Different from Text
With text, a vague prompt can still produce acceptable results. With images, precision in the prompt multiplies extraction quality.
const extractionPrompt = `Analyse this invoice and extract exactly the following fields.
Respond ONLY with valid JSON — no extra text, no markdown code blocks.
{
"invoice_number": string, // e.g. "INV-2026-00123"
"issue_date": string, // ISO format: "2026-08-12"
"supplier_name": string,
"supplier_tax_id": string, // VAT/tax number without dashes
"subtotal_eur": number, // amount excluding VAT
"vat_percentage": number, // e.g. 20
"vat_amount_eur": number,
"total_eur": number
}
Rules:
- If a field is not legible, use null (never fabricate data)
- If there are multiple dates, use the issue date, not the due date
- Amounts are numbers, never strings`;
Two techniques that materially improve production accuracy:
- Format examples in the prompt (
e.g. "INV-2026-00123") reduce format hallucinations by roughly 40% - Explicit
nullinstead of omitting the field stops the model from inventing data when a field isn't visible
Add post-extraction Zod validation to catch inconsistencies before persisting:
import { z } from 'zod';
const InvoiceSchema = z.object({
invoice_number: z.string().nullable(),
issue_date: z.string().regex(/^\d{4}-\d{2}-\d{2}$/).nullable(),
supplier_name: z.string().nullable(),
supplier_tax_id: z.string().nullable(),
subtotal_eur: z.number().positive().nullable(),
vat_percentage: z.number().min(0).max(100).nullable(),
vat_amount_eur: z.number().positive().nullable(),
total_eur: z.number().positive().nullable(),
}).refine(
(data) => {
if (data.subtotal_eur && data.vat_amount_eur && data.total_eur) {
const expected = data.subtotal_eur + data.vat_amount_eur;
return Math.abs(expected - data.total_eur) < 0.02; // rounding margin
}
return true;
},
{ message: 'Total does not match subtotal + VAT — flag for human review' }
);
This refine catches the most common extraction failure: a total that doesn't add up. Instead of writing a bad record to your ERP, the entry gets flagged for human review with the specific problem identified.
Real Cost and How to Control It
Processing a 1,200 × 1,600 px image with Claude Opus costs approximately 1,500–2,000 image tokens. At 200 invoices per month:
- Without optimisation: ~400,000 tokens × $0.015/1K = $6/month just in images
- With correct preprocessing (resize + optimised PNG): $2–3/month
The savings come from PNG file size, not from lowering resolution. Below 150 DPI the model starts making errors on small numerals — especially decimal figures, which is exactly what you don't want to get wrong in an invoice.
For high-volume clients, we add a semantic cache layer based on embeddings: documents with identical structure (same supplier, same invoice template) are served from cache without hitting the vision API.
What We Deployed in Production for a Distribution Client
A client with 400 monthly invoices from different suppliers, variable formats, some scanned from a mobile phone. The pipeline we built through their AI integration:
- Lightweight classifier — small/fast model to identify document type and relevant page count before the expensive extraction call
- Type-based preprocessing — different rules for native PDFs vs. scanned images (for scans we use 150 DPI with contrast boost in sharp)
- Zod post-extraction validation — if total ≠ subtotal + VAT, the record enters a human review queue with the specific error flagged, not as a generic "something went wrong"
- Audit trail in Supabase — every extraction stores the output JSON, processed page, token count, and confidence score for traceability and continuous improvement
Result: 94% of invoices processed without human intervention. The remaining 6% reaches the accounting team with the specific field flagged and the failure reason explained — not as a blob that needs reviewing from scratch.
If you're building a document processing pipeline and the integration timeline keeps stretching, the problem is almost always in preprocessing, not in the prompt.