Chunking strategies for RAG: from clean text to good retrieval

Chunking gets a lot of attention in RAG write-ups — overlap sizes, token counts, semantic splitters — and most of it assumes a precondition that rarely gets checked: clean, correctly-ordered text to chunk in the first place. Chunking strategy is a real lever, but it’s the second lever. The first is whether the text you’re chunking is actually right.

Three common strategies

Fixed-size chunking splits text every N tokens or characters, usually with some overlap so a sentence that straddles a boundary isn’t lost entirely. It’s simple, fast, and format-agnostic — and it’s the default in most RAG tutorials for exactly that reason.

Recursive chunking tries to split on natural boundaries first — paragraphs, then sentences, then words — only falling back to a hard character cut when a single unit is still too big. It respects sentence and paragraph structure better than a fixed-size cut, at the cost of variable chunk sizes.

Structure-aware chunking uses document structure — headings, sections, list items — as split points, so a chunk roughly corresponds to “one idea” rather than “N tokens starting at an arbitrary offset.” It produces the best retrieval quality when the structure is real, and the worst when it isn’t: a chunker that trusts heading markers on text that doesn’t actually have clean headings will split in the wrong places just as confidently as it would split in the right ones.

Why extraction quality bounds all three

Every one of these strategies operates on whatever text extraction handed it. If the source document’s extraction step interleaved two PDF columns into one garbled line, no chunking strategy — fixed, recursive, or structure-aware — recovers the original reading order. The chunker doesn’t know two columns exist; it just sees text with sentences from two unrelated ideas mashed together, and it will happily draw chunk boundaries through the middle of that mess. Structure-aware chunking is more exposed to bad extraction than fixed-size, not less, because it trusts signals (heading patterns, indentation) that a mangled extraction can fake or destroy.

This is the case for extracting the text well before you think about chunking at all. txtfetch’s PDF extraction endpoint runs Apache Tika (with Tesseract OCR for scanned pages) so multi-column reading order and table content come back as one coherent extracted_text string:

curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

{
  "status": "success",
  "extracted_text": "Executive summary\n\nQ3 revenue grew 34%..."
}

That single string is what every chunker downstream — fixed-size, recursive, or structure-aware — actually operates on. Getting it right once, upstream, is cheaper than trying to compensate for it in the chunker later.

A practical default

If you don’t have a strong reason to do something fancier: recursive chunking with a target size of 300–500 tokens and roughly 10–15% overlap is a reasonable default for most prose-heavy documents. Reach for structure-aware chunking when the source genuinely has reliable structure — technical docs with real heading hierarchies, contracts with numbered clauses — and reach for fixed-size only when you need predictable chunk counts more than you need clean boundaries (batch cost estimation, for example).

Whichever you pick, feed it into your embedding model exactly the way you already do — chunking is downstream of extraction, not a replacement for getting extraction right. If you’re wiring this into a LangChain or LlamaIndex pipeline, the document loader guide covers the integration point.

Working through a chunking strategy for your own document set? Get in touch and we’ll set you up with an API key.