Your RAG fails in prod: chunking, retrieval, reranking

Your RAG passes every test. In demo, the agent answers perfectly. You deploy to production and 30% of responses are inaccurate, incomplete, or just wrong.

The model hasn't changed. The prompt either. The problem is in the retrieval pipeline.

This isn't an edge case — it's the most repeated pattern in RAG projects we scale at DAILYMP. The same three failure points, over and over.

First failure: chunking without a strategy

Most RAG projects start the same way: split documents into fixed-size chunks (typically 512 tokens) and store them in a vector store. Fast to implement, easy to understand, catastrophic in production.

The problem isn't chunk size — it's that fixed-size chunking doesn't respect the semantic structure of the document.

Picture a 50-page product manual. Section 3.2 explains the context of a feature. Section 3.3 explains how to use it. With 512-token chunking, that split may land right in the middle of section 3.2. The resulting chunk starts mid-sentence, without the preceding context. When the retriever returns it, the LLM sees an incomplete answer.

// The most common antipattern: chunking without overlap or metadata
const chunks = splitByTokens(document, 512);
// Chunk 4 might start with:
// "...which in these cases requires enabling option B, since
//  the validation system automatically checks..."
// Without knowing which option, which cases, or which system.

The fix: mandatory overlap + structured metadata.

interface Chunk {
  text: string;
  metadata: {
    source: string;
    section: string;
    pageNumber: number;
    chunkIndex: number;
    contentType: string; // "procedure" | "definition" | "example"
  };
}

function splitWithOverlap(
  doc: Document,
  chunkSize = 512,
  overlap = 64
): Chunk[] {
  // 64-token overlap ensures each chunk shares context with the previous one.
  // Metadata enables filtering BEFORE retrieval.
}

The 64-token overlap resolves cross-boundary idea splits. Metadata enables pre-retrieval filtering: if the user asks about a specific procedure, you can restrict the search to contentType: "procedure" before computing any similarity.

Second failure: dense retrieval alone

When your knowledge base includes identifiers, proper names, or highly specific terms, semantic retrieval (cosine similarity over embeddings) fails silently.

"invoice number INV-2024-0045" and "document INV-2024-0045" have high semantic similarity with each other, but a query like "what products are on invoice INV-2024-0045?" may not retrieve the right document if the embedding doesn't capture lexical specificity well.

Dense retrieval is designed to capture meaning similarity. Not exact text matches. That's what BM25 is for.

The fix: hybrid search with Reciprocal Rank Fusion.

async function hybridSearch(
  query: string,
  vectorStore: VectorStore,
  bm25Index: BM25Index,
  topK = 10
): Promise<RankedResult[]> {
  const [denseResults, bm25Results] = await Promise.all([
    vectorStore.similaritySearch(query, topK),
    bm25Index.search(query, topK),
  ]);

  return reciprocalRankFusion([denseResults, bm25Results], topK);
}

function reciprocalRankFusion(
  resultSets: SearchResult[][],
  topK: number,
  k = 60
): RankedResult[] {
  const scores = new Map<string, number>();

  for (const results of resultSets) {
    results.forEach((result, rank) => {
      const current = scores.get(result.id) ?? 0;
      scores.set(result.id, current + 1 / (k + rank + 1));
    });
  }

  return Array.from(scores.entries())
    .sort(([, a], [, b]) => b - a)
    .slice(0, topK)
    .map(([id, score]) => ({ id, score }));
}

RRF doesn't need to normalize scores from each system (they use different scales). It combines rankings and surfaces chunks that score well in both searches. Document INV-2024-0045 may not win on semantic similarity, but it wins on BM25 — and that's enough to bring it to the top.

Third failure: top-K without threshold or reranking

Retrieval returns the K most similar chunks. But "most similar" doesn't mean "relevant enough." If your top-ten results have similarity scores between 0.35 and 0.45, you're including them in context even though none is truly relevant.

The LLM can't detect this. It sees ten chunks in context and uses all of them as if they were relevant. The result is context poisoning: responses that mix correct information with unrelated content.

The fix: score threshold + cross-encoder reranking.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

async function retrieveWithReranking(
  query: string,
  candidates: RankedResult[],
  scoreThreshold = 0.7,
  finalTopK = 4
): Promise<Chunk[]> {
  // 1. Filter by minimum threshold before reranking
  const aboveThreshold = candidates.filter(c => c.score >= scoreThreshold);

  if (aboveThreshold.length === 0) {
    return []; // No relevant results → "I don't have information on this"
  }

  // 2. Reranking with LLM (Haiku cross-encoder for cost and speed)
  const rerankPrompt = `Rank these fragments by relevance to the question: "${query}"

${aboveThreshold.map((c, i) => `[${i}] ${c.text}`).join("\n\n")}

Reply ONLY with JSON: {"ranking": [indices in relevance order], "threshold_cut": number_of_relevant_fragments}`;

  const response = await client.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 256,
    messages: [{ role: "user", content: rerankPrompt }],
  });

  const result = JSON.parse(
    response.content[0].type === "text" ? response.content[0].text : "{}"
  );

  return result.ranking
    .slice(0, Math.min(result.threshold_cut, finalTopK))
    .map((idx: number) => aboveThreshold[idx].chunk);
}

The cross-encoder evaluates each chunk in the context of the specific query, not as a standalone embedding. It can detect that a chunk about "general return policies" isn't relevant to a question about "returning a digital product purchased on promotion" — even though both contain the word "return."

The complete pipeline

These three fixes chained together give a production-grade RAG pipeline:

Query
  → Hybrid Search (dense + BM25)
  → RRF
  → Score Threshold (0.7)
  → Reranking (Haiku cross-encoder)
  → Top-4 Chunks
  → Clean Context
  → LLM

The differences from the naive pipeline:

• Overlap chunking + metadata: more precise retrieval, pre-retrieval filtering
• Hybrid search + RRF: captures semantic similarity AND exact lexical matches
• Score threshold + reranking: only information above the relevance threshold reaches the LLM

This connects directly to context engineering work: retrieval is the layer that feeds context. If retrieval fails, poisoned context reaches the LLM regardless of how good your system prompt is.

Why these failures appear late

RAG tests usually run on known questions, with known documents, under controlled conditions. In production, users ask in unexpected ways, about specific terms, against a full knowledge base.

Naive chunking works in demo because the test dataset is small and questions are calibrated. Scale to 10,000 documents and add real users, and all three failures appear at once.

At DAILYMP, we design RAG pipelines with hybrid search and reranking from the first sprint, not as a later optimization. The result: systems that don't need a refactor two months in because responses weren't precise enough.

---

Does your RAG agent work well in demo but fail with real users? Let's review it together.