Blog Details

Three weeks into the project, retrieval quality was the only open ticket that mattered, and it resisted a clean fix. The system returned the right documents, even the right pages. Yet, answers from the LLM were subtly wrong often enough that two pilot tenants independently flagged the same complaint: 'it feels like it read half the sentence.'

That complaint turned out to be literal. Our chunking strategy sliced document text at fixed character boundaries, doing exactly what we told it to do. However, a 'technically correct chunk' is not always a 'semantically useful chunk.' In our multi-tenant platform, one tenant might upload structured compliance PDFs while another uploads scanned meeting notes from a wiki. No single fixed-size rule could serve both well. This post details what we tried, what failed, and the chunking and retrieval design that ultimately held up across tenants with wildly different document structures.

Why 'Technically Correct' Retrieval Was Still Failing Users

The platform ingested documents from multiple tenants into a shared vector store, partitioned by tenant ID. Each tenant could upload any document type: contracts, internal wikis, product manuals, meeting transcripts, or spreadsheets exported as PDF. Our initial chunking used a standard fixed-size splitter, 500 tokens per chunk with a 50-token overlap, common in RAG tutorials. On paper, retrieval metrics looked fine. Cosine similarity scores were high, the correct document was almost always in the top five results, and our internal evaluation harness showed respectable recall.

The gap appeared only when we read the actual chunks a human would have to reason over. A contract's termination clause might split mid-sentence, with the condition in one chunk and the consequence three chunks away after re-ranking. A support wiki page, structured as a decision tree, flattened into chunks that lost the parent-child relationship between questions and nested sub-answers. The model wasn't hallucinating in the classic sense; it was synthesizing answers reasonably from chunks that had already lost critical information.

This is easy to miss if you only look at retrieval metrics in isolation. Recall and similarity scores measure whether you found relevant text, not whether that text is a complete, self-contained unit of meaning. A chunk can be 90% similar to the query yet still be useless if it lacks the one clause that changes the answer. We had optimized for the metric instead of the outcome, and it took reading actual failure transcripts, not dashboards, to truly see the problem.

First Attempt: Bigger Chunks and More Overlap

Our first instinct, like most teams', was to increase chunk size. If 500-token chunks lost context, perhaps 1000-token chunks with 200-token overlap would preserve enough surrounding text to mitigate boundary issues. We reasoned that overlap was cheap: it meant more storage and marginally slower indexing, but it was a straightforward lever without requiring redesign.

This helped only superficially and introduced a new, arguably worse, failure mode. Larger chunks reduced mid-clause splits but diluted the embedding. A 1000-token chunk covering three unrelated topics produced a blurry average vector, making it harder for the retriever to distinguish 'termination clauses' from 'payment terms' if both were in the same oversized block. Precision dropped even as recall held steady. For tenants with dense, multi-topic documents like policy manuals, top-k results began returning chunks that were only loosely thematic, lacking specific answers.

The increased overlap also quietly inflated our index size and per-tenant storage costs by roughly 40%, a significant factor once we priced the platform per tenant based on document volume. We had traded truncated context for diluted relevance, paying more for the privilege. At that point, we stopped treating chunk size as the primary lever. We began asking a different question: what truly defines a coherent unit of meaning in each tenant's document type, and can we chunk based on that instead of an arbitrary token count?

“A chunk can be 90% similar to the query and still be useless if it's missing the one clause that changes the answer.”

What Structure-Aware Chunking Actually Required

The real fix wasn't a smarter splitter. It was accepting that different document types required different splitting logic, determined at ingestion time rather than applied uniformly. A contract has legal structure: sections, subsections, defined terms, and clauses referencing others by number. A wiki page has hierarchical structure: headings, nested bullet lists, sometimes tables. A transcript has neither; it has speaker turns and time, with topic shifts as the only reliable, un-marked boundary.

We built an ingestion pipeline that first classified incoming documents by structural type. This used file metadata, heading detection, and a lightweight LLM pass to tag the document's dominant structure (e.g., legal, hierarchical, conversational, tabular, unstructured prose). Each structural type then routed to a different chunking strategy tuned for its natural boundaries: heading-aware recursive splitting for hierarchical documents, clause-boundary splitting anchored on numbered sections for contracts, and sliding-window splitting keyed to speaker turns for transcripts. This meant maintaining several chunking code paths, increasing engineering overhead beyond our budget, but it was the only approach that respected how each document type conveyed meaning.

The harder part wasn't writing the splitters, but deciding chunk boundaries in documents that mixed structures, which was more common than we expected. For example, a tenant's 'employee handbook' might have hierarchical headings for eighty pages, followed by an appendix of legal boilerplate with contract-style clause numbering. We chunked at the section level first, tagging each section with its detected substructure. Then, we applied the appropriate sub-strategy within that section, rather than picking one strategy for the whole document. This was more complex, but it prevented context from bleeding across structurally incompatible boundaries.

• Heading-aware recursive splitting for wikis and manuals (chunks respect top-level headings)
• Clause-anchored splitting for contracts (numbered sections and cross-references kept together)
• Speaker-turn windowing for transcripts (with topic-shift detection as a secondary signal)
• Table-preserving extraction (rows and headers kept together, avoiding fragmentation)
• Fallback recursive character splitting for unclassified content (to prevent silent ingestion failures)

The Overlooked Constraint: Retrieval Had to Stay Tenant-Scoped

Structure-aware chunking solved the semantic coherence problem, but it clashed with a constraint we had deprioritized during early design. Every retrieval call had to be strictly scoped to a single tenant's document set, with zero risk of cross-tenant leakage, even in similarity space. This security requirement was obvious and enforced from day one via metadata filtering on tenant ID. However, we hadn't fully accounted for how tenant scoping interacts with chunking strategy choices, particularly regarding shared embedding models and index structure.

Different tenants used different chunking strategies based on their document types. Consequently, the resulting chunks had varying average lengths, overlap ratios, and semantic density, even for similar source material. Feeding all this into a single shared vector index, filtered post-hoc by tenant ID, meant that similarity thresholds tuned for one tenant's chunk profile behaved differently for another's. For instance, a 0.78 cosine similarity threshold worked well for a tenant with short, clause-level chunks but returned almost nothing useful for a tenant whose chunks were longer, topic-window transcripts. The embedding space geometry simply wasn't comparable across these chunking strategies.

We resolved this by moving from a single global similarity threshold to per-tenant calibrated thresholds. These were computed during onboarding using a small labeled set of query-answer pairs specific to that tenant's document corpus. We also reconsidered index partitioning. Instead of one large filtered index, we moved toward tenant-namespaced sub-indices within the same vector store. This allowed us to tune retrieval parameters, including chunk size defaults and re-ranking behavior, independently per tenant, preventing one tenant's document quirks from degrading another's results. This added operational overhead, as onboarding a new tenant now involved a calibration step beyond just an upload-and-embed pipeline. However, it was the only way to make chunking strategy and tenant isolation work together.

Measuring the Difference: Before and After Structure-Aware Chunking

We evaluated the change using eighty real support queries from pilot tenant usage. These queries were split roughly evenly across the three dominant document structures in our corpus: legal/contractual, hierarchical/wiki, and conversational/transcript. Each query had a human-graded 'correct and complete' answer determined beforehand, independent of the system being tested, ensuring unbiased grading. We measured both retrieval precision at the chunk level and end-to-end answer correctness after generation, recognizing that these two numbers had been diverging.

The results made the case better than any argument. Fixed-size chunking with generous overlap looked competitive on raw retrieval precision, missing the mark mostly on legal documents where clause splitting caused the most damage. However, the end-to-end correctness gap was much larger than the retrieval gap alone suggested. A partially-context chunk doesn't just fail to help the model; it actively misleads it into confident, wrong synthesis. Structure-aware chunking closed most of that gap. The improvement was largest exactly where we expected: legal and hierarchical documents, where boundary semantics carry the most meaning per token.

What surprised us was the transcript category, where structure-aware chunking helped less than expected. Speaker-turn windowing improved upon fixed-size splitting, but conversational data is inherently messier. Topics drift gradually rather than at clean boundaries, and no chunking strategy fully solved this. This result was useful: it showed us that chunking has a ceiling, and for conversational data, the greater lever was query rewriting and multi-hop retrieval, not just chunk boundary logic.

Chunking Is a Modeling Decision, Not a Preprocessing Step

The single biggest mindset shift this project forced on us was treating chunking as part of the retrieval model itself, not merely a preprocessing detail configured once and forgotten. Teams new to RAG, including our own team six months earlier, often treat chunk size as a hyperparameter to sweep once during a proof of concept, pick a reasonable default, and then move on to prompt engineering or model selection, where perceived leverage is higher. In a single-tenant system with homogeneous documents, that shortcut can survive for a long time without visible harm. In a multi-tenant system with heterogeneous document structures, however, it fails quickly and in ways hard to diagnose from metrics alone.

The deeper lesson is that chunk boundaries encode an assumption about where meaning lives in a document. This assumption is only valid if it matches how the document was authored to be read. A contract's meaning lives in its clause structure. A wiki's meaning lives in its heading hierarchy. A transcript's meaning lives in conversational turns and topic continuity. Fixed-size chunking implicitly assumes meaning is uniformly distributed across characters, which is almost never true. This failure only becomes visible with enough document diversity to expose the assumption's limits.

For anyone building a document RAG platform meant to serve more than one type of customer, we would now treat structural classification and tenant-aware calibration as first-class architectural components from day one. These are not fixes bolted on after a pilot tenant complains. It costs more upfront in engineering time and ongoing calibration effort. However, the alternative is shipping a system that looks correct on every dashboard while quietly giving users worse answers than a simple keyword search. That is a much harder failure to detect and a much worse one to explain to a customer.

• Chunking assumes where meaning lives; validate this assumption per document type.
• Retrieval precision and end-to-end answer correctness can diverge; measure both.
• Tenant isolation is a retrieval-tuning boundary, not just security, when chunking strategies diverge.
• Some document types (e.g., conversational data) have a chunking ceiling; focus on query handling, not just chunk size.
• Structural classification at ingestion is cheaper to build early than to retrofit later.