Chunking strategies for RAG: from clean text to good retrieval

Chunking gets a lot of attention in RAG write-ups: overlap sizes, token counts, semantic splitters. Most of it assumes a precondition that rarely gets checked. The text needs to be clean and in the right order before you chunk it. Chunking strategy is a real lever, but it’s the second lever. The first is whether the text you’re chunking is actually right.

Three common strategies

Fixed-size chunking splits text every N tokens or characters, usually with some overlap so a sentence that straddles a boundary isn’t lost entirely. It’s simple, fast, and format-agnostic. That’s why most RAG tutorials use it by default.

Recursive chunking tries to split on natural boundaries first: paragraphs, then sentences, then words. It falls back to a hard character cut only when a single unit is still too big. It respects sentence and paragraph structure better than a fixed-size cut. The cost is variable chunk sizes.

Structure-aware chunking uses document structure as split points: headings, sections, list items. A chunk then matches one idea, not N tokens starting at an arbitrary offset. This produces the best retrieval quality when the structure is real. It produces the worst when the structure isn’t real. A chunker that trusts heading markers on text without clean headings will split in the wrong places. It does this just as confidently as it splits in the right ones.

Why extraction quality bounds all three

Every one of these strategies operates on whatever text extraction handed it. Say the source document’s extraction step interleaved two PDF columns into one garbled line. No chunking strategy recovers the original reading order then, not fixed, recursive, or structure-aware. The chunker doesn’t know two columns exist. It just sees text with sentences from two unrelated ideas mashed together. It will happily draw chunk boundaries through the middle of that mess. Structure-aware chunking is actually more exposed to bad extraction than fixed-size, not less. That’s because it trusts signals like heading patterns and indentation, and a mangled extraction can fake or destroy those signals.

This is the case for extracting the text well before you think about chunking at all. txtfetch’s PDF extraction endpoint runs Apache Tika, with Tesseract OCR for scanned pages. Multi-column reading order and table content come back as one coherent extracted_text string:

curl -X POST "https://api.txtfetch.com/v1/extract?url=https://example.com/whitepaper.pdf" \
  -H "Authorization: Bearer $TXTFETCH_KEY"

{
  "status": "success",
  "extracted_text": "Executive summary\n\nQ3 revenue grew 34%..."
}

That single string is what every chunker downstream operates on, whether fixed-size, recursive, or structure-aware. Getting it right once, upstream, is cheaper than trying to compensate for it later in the chunker.

A practical default

If you don’t have a strong reason to do something fancier, use recursive chunking. A target size of 300-500 tokens with roughly 10-15% overlap is a reasonable default for most prose-heavy documents. Reach for structure-aware chunking when the source genuinely has reliable structure, such as technical docs with real heading hierarchies or contracts with numbered clauses. Reach for fixed-size chunking only when you need predictable chunk counts more than clean boundaries, for example when estimating batch cost.

Whichever you pick, feed it into your embedding model the way you already do. Chunking is downstream of extraction. It’s not a replacement for getting extraction right. If you’re wiring this into a LangChain or LlamaIndex pipeline, the document loader guide covers the integration point.

Want to see this argument on your own text instead of taking it on faith? The chunk previewer runs all three strategies above against text you paste in. It shows the resulting boundaries, token estimates, and overlap. It also scans for specific extraction-damage signals that wreck good chunking: hyphenated line breaks, mojibake, collapsed tables. It’s free and runs entirely in your browser. It also includes a clean-vs-naive-PDF-extraction sample of the same source page, so you can see the difference directly.

Working through a chunking strategy for your own document set? Get in touch and we’ll set you up with an API key.