In this article, you will learn how to choose the right chunking strategy for a retrieval-augmented generation system by working through a structured decision tree tied to the properties of your documents and retrieval task.
Topics we will cover include:
- What each major chunking strategy assumes about your documents, and when those assumptions hold.
- A six-question decision tree that maps concrete document and retrieval conditions to specific chunking strategies.
- The failure signals that indicate a chosen strategy is not working, and the fixes that tend to resolve them.
Introduction
When you are building a retrieval-augmented generation system, you often work with documents that are larger than what an embedding model or LLM context window can accommodate, so you need to split them into smaller chunks. Chunk size and overlap may seem like straightforward configuration choices, but the way you set those boundaries directly affects retrieval. Poor boundaries can cause retrieval to miss the passage that answers a query, or return a chunk that is technically relevant but lacks the surrounding context needed to make it useful.
A chunking strategy defines how you decide where to split a document, and that choice can have a major effect on retrieval quality. When the boundaries preserve the right context, embeddings, vector search, and generation have a stronger basis to work with. When the boundaries break important context, reranking and prompt tuning cannot fully recover what was lost during splitting. No single chunking method works well for every document. The right choice, therefore, depends on the structure of the document and the retrieval task.
This article covers:
- What each major chunking strategy assumes about your documents
- A decision tree that maps every one of those strategies to a concrete condition worth checking
- How the strategies land on real document types, with tradeoffs laid out side by side
- The failure signals that tell you a chosen strategy isn’t working
We start with why this choice is important.
What Each Chunking Strategy Assumes
Each chunking strategy is based on a specific assumption about the document. Choosing a strategy that matches the document’s structure and needs helps improve retrieval quality while avoiding unnecessary complexity:
Strategy
What It Assumes
Fixed-size chunking
Boundary placement doesn’t matter much, since overlap can patch whatever gets cut mid-thought
Recursive chunking
Paragraph and sentence boundaries are a safe enough proxy for meaning when no stronger structure is available
Document-based chunking
The document’s own structure — headers, sections, function boundaries — lines up with its semantic structure
Semantic chunking
Topic shifts are detectable sentence by sentence through embedding distance
Late chunking
A chunk’s meaning depends on the whole document, not just its own text
Adaptive chunking
A single chunk size can’t serve sections that vary sharply in information density
Hierarchical chunking
Retrieval needs to answer both broad and specific questions from the same corpus
LLM-based chunking
Only a model reading the text can find logical breakpoints that structure and embeddings miss
Agentic chunking
The collection is varied enough that no single rule handles all of it well
When the strategy fits the document, its added complexity can be worthwhile. When it does not, it adds overhead without improving retrieval. For example, semantic chunking is unnecessary for a short FAQ entry, while fixed-size chunking can split related clauses in a contract. These mismatches can lead to retrieval problems that originate in preprocessing rather than generation.
Working Through a Decision Tree to Choose a Chunking Strategy
The tree has six branching questions, each narrowing the strategy space based on a concrete property of the document or the retrieval task. Every question below ends with a direct answer: which strategy to use if the condition holds, and where to go next if it doesn’t. Following them in order helps you choose a strategy based on the document and retrieval task rather than defaulting to a familiar option.
Question 1: Does the Document Need Chunking at All?
Short, self-contained content — such as FAQ entries, product descriptions, or single config snippets — often fits comfortably inside one embedding call and covers exactly one topic.
If the document sits well within the model’s context window and doesn’t mix subjects, embed it whole and skip chunking. Splitting content that is already coherent only adds retrieval overhead. If the document is too long, or covers more than one topic, move to Question 2.
Question 2: Does the Document Have a Clear Internal Structure?
Some documents already contain natural boundaries that reflect how their content is organized. Markdown headings, HTML elements, PDF sections and tables, and code functions or classes can all provide meaningful chunk boundaries.
If the document has a reliable structure that maps to logical content units, use document-based chunking. The document is parsed according to its format, keeping sections, paragraphs, tables, functions, or other structural elements together. If the structure is inconsistent, incomplete, or doesn’t provide useful boundaries, move to Question 3.
Question 3: Do Chunks Need Context From Elsewhere in the Document?
Some documents carry dependencies that span the whole text. Take, as an example, a legal contract that defines a term in section one and uses it in section eight, or a technical spec whose terminology only makes sense next to an earlier paragraph.
If retrieval needs to preserve that document-wide relationship, use late chunking. Embed the full document with a long-context model first, then derive chunk-level embeddings from that result, so each chunk still carries a trace of the whole document. If chunks can stand on their own without losing meaning, move to Question 4.
Question 4: Does Information Density Vary, or Do You Need Multiple Levels of Granularity?
This question covers two different conditions, each pointing to a different strategy:
- If information density varies across sections, use adaptive chunking to dynamically adjust parameters such as chunk size and overlap to match the content. Dense, information-rich sections can use smaller, more granular chunks, while broader or less dense sections can use larger chunks.
- If the same corpus needs to support both broad questions (such as what a document covers) and specific questions (such as exact syntax), use hierarchical chunking, with coarse parent chunks and finer-grained child chunks.
If neither condition applies, move to Question 5.
Question 5: Does Retrieval Quality Matter More Than Cost and Latency?
This is the cost and quality tradeoff. If retrieval quality is the priority and the budget supports a model call per document, use LLM-based chunking to identify logical breakpoints or atomic propositions. If quality still matters but per-document model calls are too costly, and topic shifts are unpredictable, use semantic chunking based on sentence-embedding distance. It is less costly than an LLM pass while still accounting for meaning. If speed and cost are the priority, move to Question 6.
Question 6: Is This One Document Type, or a Heterogeneous Collection?
If the corpus mixes document types where the right strategy differs from one document to the next, use agentic chunking. For example, a collection might contain structured reports, narrative transcripts, and code. An agent can inspect each document and select the appropriate approach.
If the collection is reasonably uniform, recursive chunking is a practical default. It splits content at paragraph, sentence, and then word boundaries until the chunks fit the target size, while preserving natural structure without requiring per-document analysis.
If speed matters more than chunk quality, fixed-size chunking is a reasonable fallback for quick prototypes. After working through these questions, the decision tree should look like so:
Mapping the Decision Tree to a Chunking Strategy
Working through the tree lands you on one of the strategies below. Systems often blend more than one, but the table maps each strategy to the situation it fits best and the cost that comes with it:
Strategy
Best For
Key Tradeoff
No chunking
Short, single-topic content that already fits the context window
None — skips preprocessing entirely
Document-based chunking
Content with reliable structural markup (Markdown, HTML, code)
Breaks down when headers don’t track topic boundaries
Late chunking
Documents with cross-references or dependencies spanning sections
Requires a long-context embedding model
Adaptive chunking
Corpora with sharply uneven information density
Added indexing and retrieval complexity
Hierarchical chunking
Corpora needing both broad and specific queries answered well
Two levels of chunks to build and keep in sync
LLM-based chunking
Complex, high-stakes documents where retrieval quality justifies the cost
Highest compute cost per document
Semantic chunking
Documents with unpredictable topic shifts on a moderate budget
Needs threshold tuning to avoid inconsistent chunk counts
Agentic chunking
Heterogeneous collections where the optimal strategy varies by document
Requires per-document analysis before chunking begins
Recursive chunking
General-purpose default for a reasonably uniform corpus
Less semantically precise than meaning-aware methods
Fixed-size chunking
Fast prototyping when speed matters more than chunk quality
Ignores structure and meaning entirely
The right chunking strategy depends on the structure, complexity, and retrieval needs of your data, with the goal of balancing retrieval quality, computational cost, and implementation complexity.
Recognizing Common Chunking Pitfalls
A strategy that looked right on paper can still misbehave once it meets actual documents. The table below lists the signals worth watching for, what they usually mean, and the fix that tends to work:
Symptom
Likely Cause
Fix
Chunks consistently cut mid-sentence or mid-table
A structure-blind method applied to a structured document
Switch to recursive chunking instead of widening overlap
Semantic chunking produces wildly inconsistent chunk counts across similar documents
Similarity threshold is miscalibrated
Tighten the threshold, or fall back to LLM-based boundaries for the documents that keep misbehaving
Document-based chunks range from one line to several pages
Headers don’t reliably track topic boundaries
Combine document-based splitting with a recursive pass inside oversized sections
Hierarchical or adaptive chunking adds latency without improving answers
Corpus is small and fairly uniform
Drop back to recursive chunking
Agentic chunking blows past its compute budget on a large corpus
Per-document analysis is repeated on every run instead of cached
Cache strategy decisions, or run the analysis at the collection level rather than per document
These pitfalls can help you identify when a chunking strategy needs to be adjusted for better retrieval and efficiency.
Wrapping Up
Choosing a chunking strategy starts with understanding the structure of your documents and the retrieval task they need to support. Factors such as document structure, context dependencies, information density, retrieval requirements, and cost all shape that choice.
Use the decision tree to identify a suitable starting point, then validate it with your own queries and retrieval results. Start with the simplest strategy that fits your needs, and introduce more advanced approaches only when they address a specific retrieval challenge.
The best results come from matching chunk boundaries to the information you need to retrieve, while keeping the added complexity proportional to the value it provides.