By
Sept. 9, 2026
Your RAG system returns confident answers that are subtly, dangerously wrong. The retrieval logs show relevant chunks surfaced. The LLM is state-of-the-art. The embeddings model sits at the top of the leaderboard. Yet when your VP of Sales asks the AI assistant about return policies for Product A versus Product B, it blends details from both into a single fabricated answer that could cost the company a client relationship—or worse, regulatory exposure.
The culprit isn't the model. It's the line of code most teams write once and never revisit:
split_text(text, chunk_size=1000, overlap=200)
Ship it. That's the entire strategy. And it's silently destroying your enterprise RAG implementation.
Most production RAG systems still rely on what the research community now calls "naive chunking"—splitting documents at arbitrary character or token boundaries with a fixed overlap window. Tutorials teach it. Quickstart guides default to it. It's simple, fast, and catastrophically inadequate for enterprise documents.
The data is stark. Systems using naive chunking face baseline hallucination rates around 20 percent, according to recent production failure analyses. One documented case showed a 78X improvement in accuracy simply by moving from character-count splitting to structure-aware chunking. The problem isn't that chunks are the wrong size. It's that a chunk, read in isolation, becomes ambiguous about what it refers to.
Consider a real-world failure pattern: You have 100 products, each with different return policies. The policies are lengthy—requiring chunking to fit within embedding model limits and LLM context windows. Naive chunking splits "The notice period is 90 days" from the heading "Product A: Enterprise License" and places it three chunks away from "Product B requires 30-day written notice." When a user asks about Product A's return terms, the retrieval system surfaces both fragments. The LLM, seeing no clear boundaries, synthesizes an answer that's neither.
This isn't a retrieval ranking problem. It's not an embedding quality issue. The chunks themselves have lost the context that made them interpretable.
Enterprise documents have structure: policies with numbered clauses, API documentation with code blocks and parameter tables, SharePoint knowledge bases organized by headings, legal contracts with nested sections, product catalogs with specifications. Fixed-size chunking treats all of this as an undifferentiated character stream.
Three production failure modes emerge consistently:
Boundary violations. A 1,000-character window splits a pricing table vertically, separating column headers from their values. The chunk containing "$299/month" no longer knows which tier or feature set it describes. Retrieval surfaces it. The LLM confidently assigns the price to the wrong product.
Context collapse. A policy document chunk begins mid-paragraph with "However, exceptions apply when..." The preceding context explaining what the baseline rule is lives in the previous chunk. Alone, the exception is meaningless—or worse, gets interpreted as a general rule.
Semantic fragmentation. A five-step procedure gets split across three chunks. When a user asks how to complete the process, retrieval returns steps 2, 4, and 5. The LLM fills in the gaps with plausible-sounding but incorrect interpolations.
The research is unequivocal: default 500-token chunking strategies fail on regulated text, often silently. Failures go unnoticed because teams deploy RAG systems without measuring retrieval precision and recall. Quality degrades over time. Improvements become difficult because there's no baseline.
Even when teams move beyond naive chunking, they often miss the second failure point: chunks without metadata are ambiguous by design.
A chunk that reads "Eligible employees receive 15 days annually" is perfectly retrievable. The embedding captures the semantic meaning. But which employees? Which policy year? Which division? If your knowledge base contains HR policies for three subsidiaries across four countries, that chunk could match any of twelve different contexts.
The fix isn't a better chunker. It's understanding your document structure first, then enriching every chunk with structured attributes: document title, section heading, subsection, policy effective date, jurisdiction, product SKU, whatever taxonomy makes a chunk interpretable when read alone.
Metadata-enriched chunking moves recall more than swapping embedding models, especially for policies, API docs, tables, and multi-step procedures. One systematic framework study showed that dynamic metadata generation using LLMs—adding titles, summaries, keywords, and entity tags to each chunk—significantly improved semantic understanding and retrieval accuracy in enterprise settings. The cost is one-time, typically pennies per document when using prompt caching.
Yet most production systems still chunk first, embed second, and never look back.
The solution emerging from production deployments is deceptively simple: chunk based on the document's inherent structure, not arbitrary token counts.
For a SharePoint policy document, the correct chunk is everything under a single H2 heading—the full content that belongs to one logical section, with the heading itself preserved in metadata. For API documentation, chunk by endpoint: the method signature, parameters, request/response examples, and error codes as a single unit. For legal contracts, chunk by numbered clause. For product catalogs, chunk by SKU.
This approach, sometimes called "document-aware chunking" or "layout-aware chunking," respects semantic boundaries that already exist. It produces variable-length chunks—some 200 tokens, others 1,500—but each chunk is coherent and self-contained.
Implementation varies by document type. Markdown and HTML have explicit structure you can parse. PDFs require layout analysis; vision-capable models can identify that a block is a table and keep it whole, preserving relationships a token counter destroys. Unstructured text—narrative prose, meeting transcripts—still needs size-based splitting, but even there, sentence and paragraph boundaries outperform mid-word cuts.
The performance delta is measurable. Benchmarks comparing five chunking strategies—fixed-size, recursive, semantic, document-aware, and sliding window—show document-aware approaches consistently deliver higher retrieval accuracy on enterprise corpora, particularly for structured and semi-structured content.
For complex enterprise use cases, three advanced strategies are gaining production adoption:
Hierarchical (parent-child) chunking creates a two-tier system. Small chunks (150-300 tokens) are embedded for precise retrieval. But each small chunk links to a larger parent section (1,000-2,000 tokens). Retrieval uses the small chunks to find relevance, then returns the full parent context to the LLM. This solves the "correct chunks retrieved but LLM responses lack depth" problem. The tradeoff: increased index size and added complexity in retrieval logic.
Sliding window with overlap moves a defined window across text, creating overlapping segments. Each chunk shares 20-50 percent of its tokens with neighbors, ensuring information near boundaries appears in multiple chunks. This improves retrieval consistency but increases storage and token costs. It's effective for continuous narrative where ideas span natural break points, less useful for structured documents where section boundaries are clear.
Late chunking and contextual retrieval attack the context-loss problem from opposite directions. Late chunking, introduced by Jina AI in 2024, embeds the entire document first, then pools embeddings per chunk, preserving cross-chunk context in the vector representation itself—zero LLM calls, pure embedding-model technique. Contextual retrieval, popularized by Anthropic, generates a brief context summary for each chunk using an LLM pass ("This chunk is from the Product A return policy section..."), then prepends it before embedding. Both techniques measurably reduce ambiguity. The choice hinges on cost profile: late chunking has no per-chunk LLM cost but requires embedding-model support; contextual retrieval works with any embedder but incurs one-time LLM inference cost per chunk.
The recurring theme in enterprise RAG failures is the absence of metrics. Teams measure end-to-end answer quality—if they measure at all—but don't isolate retrieval performance.
Two metrics matter:
Recall@k: Does at least one relevant chunk appear in the top-k retrieved candidates? If recall is low, the truth never entered the system. The problem is chunking, embedding model choice, or index coverage—not the LLM.
Precision@k: What fraction of the top-k chunks are actually relevant? If you retrieve 20 chunks but only 3 matter, precision is low. The LLM wastes context window on noise, and answer quality suffers.
Measuring these requires a labeled evaluation set: queries with known ground-truth chunks. Building one is tedious but essential. Without it, you're optimizing blind.
Production teams that instrument retrieval separately from generation can pinpoint failures. When logs show retrieval returns relevant chunks outside the top-5 results, add a reranker. When correct chunks surface but answers lack depth, implement hierarchical chunking. When chunks are ambiguous, enrich metadata. When boundaries split context, switch to document-aware splitting.
Each fix is targeted. Each is measurable.
If your enterprise RAG system is underperforming, resist the urge to swap LLMs or chase the latest embedding model. Audit your chunking strategy first.
Ask:
For most enterprises, moving from naive chunking to document-structure-aware chunking with metadata enrichment will deliver more improvement than any model upgrade. The work is unsexy—parsing document layouts, tagging sections, instrumenting retrieval logs—but it's the foundation that makes everything else work.
The LLM can only be as good as the context you give it. And the context you give it is determined entirely by how you chunk.
Start with a retrieval audit. Take 20-30 representative queries your users actually ask. For each, manually identify which chunks should be retrieved. Run them through your current system and measure recall@5 and precision@5. If recall is below 80 percent, your chunking strategy is losing critical context before the LLM ever sees it.
Then examine your document types. If you're chunking structured content—policies, API docs, tables, contracts—with fixed token counts, you're leaving 10X improvements on the table. Implement document-aware chunking for your three most-queried document types. Measure again.
Enrich metadata incrementally. Start with document title and top-level section heading for every chunk. Add taxonomy tags (product name, policy type, jurisdiction) where applicable. Use an LLM pass to generate contextual summaries if ambiguity persists.
Instrument everything. Log every retrieval, every chunk returned, every query. Build dashboards that surface retrieval failures separately from generation failures. Make chunking strategy a first-class concern in your RAG architecture, not a preprocessing afterthought.
The enterprises winning with RAG aren't using better models. They're using better chunking. The gap between a functional RAG system and one that's trusted enough to put in front of customers or compliance officers comes down to whether a chunk, read alone, means what it should. Fix that, and the LLM will do the rest.
Reckonsys Team
Authored by our in-house team of engineers, designers, and product strategists. We share our hands-on experience and practical insights from the front lines of digital product engineering.
Let's collaborate to turn your business challenges into AI-powered success stories.
Get Started
