Fine-tuning, RAG or Prompting: Pick Wrong, Lose Weeks

There's a decision that teams starting with AI agents get wrong at an alarming rate: they choose the model adaptation architecture before understanding what problem they actually have.

Some start with fine-tuning because they think "customizing the model" is the right move. Others go straight to RAG because "we need the agent to know our documents." Very few ask whether well-built prompting is already enough.

The result is always the same: weeks of work, several thousand euros, and a system that doesn't perform noticeably better than a simpler solution would have.

This article is the guide that should exist before making that decision.

Three Approaches, Three Different Contexts

Prompting (in-context learning and few-shot) uses the model as-is, giving it instructions and examples in the context of each call. You don't modify the model — you just communicate with it better.

RAG (Retrieval-Augmented Generation) connects the model to an external knowledge base — a vector store, a database, documents — that retrieves relevant information in real time before generating the response. The model doesn't change; what it knows at inference time does.

Fine-tuning retrains the model on domain-specific examples. The model learns new patterns and behaviors. The result is a modified model you deploy in place of the base one.

None of these is universally superior. They're answers to different problems, with different costs and complexity. The mistake is applying the most sophisticated solution without first checking whether the simplest one works.

When Prompting Is Enough (And Why Most Teams Abandon It Too Early)

Most production agents we see with behavior problems have them because of poorly constructed prompts, not model limitations.

A model like Claude Sonnet or GPT-4o, well instructed, can:

• Classify documents with high precision if you give it clear examples in the prompt
• Answer questions about a specific domain if you provide the right context
• Generate structured JSON outputs if you use a precise instruction and an explicit schema
• Follow complex decision flows if you describe them clearly

The limit of prompting isn't response quality — it's the context window. If your agent needs to process or remember more information than fits in a context window (typically 128k–200k tokens), pure prompting doesn't scale.

// Example: classifier agent with few-shot in TypeScript
const systemPrompt = `You are a support ticket classifier.
Valid categories: BILLING, TECHNICAL, ACCOUNT, OTHER.

Examples:
Input: "I can't access my account since yesterday"
Output: { "category": "ACCOUNT", "priority": "HIGH" }

Input: "When does my plan renew?"
Output: { "category": "BILLING", "priority": "LOW" }

Always return valid JSON with category and priority.`;

const response = await anthropic.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 200,
  system: systemPrompt,
  messages: [{ role: "user", content: ticketText }]
});

When to escalate: when the agent needs to know information that doesn't fit in the prompt (extensive documentation, customer history, product catalogs) or when knowledge changes frequently.

When to Use RAG — And the Chunking Error That Breaks It

RAG is the right answer when the agent needs to access knowledge that doesn't fit in context or that updates frequently. Real examples:

• A support agent answering based on your 2,000-page internal documentation
• An assistant that knows the full interaction history with each customer
• A sales agent querying a live, updated product catalog

Standard implementation in TypeScript with Supabase and pgvector:

// Semantic retrieval with basic reranking
async function retrieveContext(query: string, topK = 5): Promise<string> {
  const embedding = await generateEmbedding(query);

  const { data: chunks } = await supabase.rpc("match_documents", {
    query_embedding: embedding,
    match_threshold: 0.75,
    match_count: topK * 2  // fetch more, then filter
  });

  // Basic reranking by lexical overlap
  const reranked = chunks
    .map(chunk => ({
      ...chunk,
      score: chunk.similarity + keywordOverlapScore(query, chunk.content)
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);

  return reranked.map(c => c.content).join("\n\n---\n\n");
}

The mistake nobody mentions: chunking. Most teams chunk by fixed size (every 500 tokens) without respecting the semantic structure of the document. The result: chunks that cut mid-argument, lose context, and generate partial or wrong answers.

The practical rule: chunk by semantic unit (paragraphs, H2 sections, numbered steps) with 10–15% overlap between consecutive chunks. For technical documents, prepend metadata (section title, page number) to the start of each chunk.

The RAG architecture in production is far more delicate than tutorials suggest: chunking, embedding, retrieval, reranking, and context assembly are five independent failure points that need to be tuned together.

Fine-Tuning: When It Makes Sense and When It's an Expensive Mistake

Fine-tuning is the right choice in a narrower set of cases than most people assume:

• The model needs to adopt a very specific writing style or tone that can't be replicated through prompting
• You have thousands of labeled examples of correct inputs/outputs for your use case
• The behavior you need is consistently different from the base model's default
• Inference cost is critical and you need a smaller model to behave like a larger one for your specific task

What fine-tuning does not solve:

• The model knowing things not in its training data (that's what RAG is for)
• The model being accurate on data that changes (fine-tuning is static)
• Hallucination (fine-tuning may reduce it, but won't eliminate it)

The real cost of a well-done fine-tune includes: dataset preparation and cleaning, training iterations, evaluation against a custom benchmark, and maintenance when the base model version changes or your task evolves. Before starting, ask yourself whether prompting + RAG achieves the same result with 10% of the effort.

// When to use each — decision tree in code
type LLMStrategy = "prompting" | "rag" | "fine-tuning";

function selectStrategy(requirements: {
  knowledgeSize: "small" | "medium" | "large";
  knowledgeChanges: boolean;
  dataAvailable: number; // labeled examples
  behaviorIsSpecific: boolean;
}): LLMStrategy {
  const { knowledgeSize, knowledgeChanges, dataAvailable, behaviorIsSpecific } = requirements;

  // Always try prompting first
  if (knowledgeSize === "small" && !knowledgeChanges) return "prompting";

  // If knowledge is large or changes, use RAG
  if (knowledgeSize === "large" || knowledgeChanges) return "rag";

  // Fine-tuning only if RAG+prompting don't produce the specific behavior
  // and you have enough data
  if (behaviorIsSpecific && dataAvailable > 1000) return "fine-tuning";

  // Default: RAG handles more cases than fine-tuning
  return "rag";
}

The Right Pattern: Progressive Escalation

The architecture that works in production is not choosing one of the three: it's starting with the simplest and escalating only when there's evidence it's not enough.

1. Start with prompting — build the agent with few-shot examples and a precise system prompt. Measure quality against real test cases.
2. If context doesn't reach: add RAG — connect the vector store to the documents or data the agent needs. This complements prompting, not replaces it.
3. If behavior is still insufficient after iterating: evaluate fine-tuning — with a dataset of at least 500–1000 examples, a separate evaluation set, and clear success metrics.

In the AI integration and automation agent projects we implement, more than 70% of cases are solved with well-built prompting + RAG. Fine-tuning is rare, and when it appears it's because the use case is very specific and the data volume justifies it.

Real Results

The most expensive mistake we've seen in teams that come to us: six weeks of fine-tuning for a support agent that kept answering incorrectly. The real problem was documentation chunking and an ambiguous system prompt. Improved prompting + RAG with correct chunking → 92% accuracy in two days of work.

Not because fine-tuning is bad. But because it wasn't the right tool for that problem.

Architecture Decisions Are Technical Debt Before You Write a Line of Code

Choosing the wrong architecture isn't just a technical problem — it's debt that compounds. A system built on fine-tuning when RAG was enough needs retraining every time knowledge changes. A system in RAG when prompting was sufficient carries unnecessary operational complexity.

The correct evaluation takes a day. The cost of choosing wrong can be a month.

If you're designing your AI agent architecture and want someone with real production experience to review it before you commit weeks of work, let's talk through your case.

We'll review your architecture before you start →