Before documents can be embedded, they are split into chunks. How big those chunks are is one of the most important decisions in the whole pipeline, because a chunk is the unit that gets matched and the unit that lands in the prompt.
The size trade-off#
Both directions have a cost.
- Too big: a chunk matches a wider range of queries, but it adds noise when it lands in the prompt. The model has to sift through unrelated text, and you pay for more tokens.
- Too small: a chunk matches precisely, but it loses context. "Revenue grew 3% in Q2" tells the model almost nothing without knowing which company, which year, and what the surrounding discussion was about.
| Chunk size | Matching | In the prompt |
|---|---|---|
| Too big | Matches broadly, less precisely | Noisy, more tokens |
| Too small | Matches precisely | Missing context |
Overlap#
Overlap partly fixes the too-small problem. When you split, adjacent chunks share a few tokens at their edges. A key sentence near a boundary then appears in both chunks instead of being cut in half.
A typical starting point is 256 to 512 tokens per chunk with 20 to 50 tokens of overlap. The right size depends on your documents and your queries, so treat these as defaults to test, not rules.
def chunk(tokens: list[str], size: int = 400, overlap: int = 40) -> list[list[str]]:
# Each chunk starts `size - overlap` tokens after the previous one,
# so neighbors share `overlap` tokens at their edges.
step = size - overlap
return [tokens[i:i + size] for i in range(0, max(len(tokens) - overlap, 1), step)]Smarter chunking strategies#
Fixed size with overlap is the baseline, but it ignores the structure of the document. It can cut a sentence, a table row or a function in half. The other strategies respect more of that structure, at the cost of more work.
| Strategy | How it splits | Trade-off |
|---|---|---|
| Fixed size + overlap | Every N tokens | Simple, ignores structure |
| Recursive | Paragraphs first, then sentences, then words | Respects sentence boundaries |
| Structure-aware | Markdown headings, HTML tags, code functions, table rows | Coherent chunks, needs consistent structure |
| Semantic | Embeds each sentence and splits where meaning shifts | One idea per chunk, slowest to build |
Small-to-big retrieval sidesteps the size trade-off. You index small chunks for precise matching, but return the larger parent section to the model, so the match is precise and the context is rich. The weak spots note shows how it works.
Where to start#
Do not reach for the most advanced strategy first:
- Start with recursive splitting.
- Move to structure-aware chunking if your documents have a consistent structure.
- Evaluate before you optimize further.
Short line#
Chunks that are too big add noise and cost; chunks that are too small lose context. Start around 256 to 512 tokens with 20 to 50 tokens of overlap, use recursive splitting, and move to structure-aware chunking or small-to-big retrieval when evaluation shows the need.
Key takeaways#
- Chunk size is one of the most important RAG decisions.
- Too big: noisy prompts and more tokens. Too small: precise matches with missing context.
- Overlap lets a sentence on a boundary appear whole in at least one chunk.
- Start around 256 to 512 tokens with 20 to 50 tokens of overlap, then test.
- Recursive, structure-aware and semantic chunking respect more structure, at more cost.
- Small-to-big retrieval gives a precise match with rich context.