What chunk size should I use for RAG?

Recursive chunking with a target of 300–500 estimated tokens and 10–15% overlap is a reasonable default for most prose-heavy documents. It's the same default the chunking-strategies guide recommends. Reach for structure-aware chunking when the source has reliable headings, like technical docs or contracts with numbered clauses. Use fixed-size chunking only when you need predictable chunk counts more than clean boundaries.

How much overlap should chunks have?

10–15% is a reasonable starting point. That's enough for a sentence straddling a boundary to still appear in full in at least one chunk. It isn't so much that you pay to embed and store the same text repeatedly. Overlap matters more for fixed-size chunking, which ignores sentence boundaries entirely. Recursive and structure-aware chunking mostly avoid cutting mid-sentence in the first place, so overlap matters less for them.

Why are my chunks cut off mid-sentence?

Either you're using fixed-size chunking, which cuts at a hard character window regardless of what it lands on. Or a single sentence in your source text is longer than your target chunk size. Recursive and structure-aware chunking both fall back to a word-level split when even one sentence alone overflows the target. There's no smaller boundary above a hard character cut. This tool flags every chunk that starts or ends mid-sentence, so you can see exactly where and why.

Does this tool upload my text anywhere?

No. Chunking and the extraction-quality scan both run entirely in your browser. Nothing you paste, load, or drop is sent over the network. The only thing that leaves your browser is the JSON report, if you choose to copy it.

How many tokens is my document, really?

This tool can only estimate. It counts letter/digit runs and punctuation runs and applies a documented ~4-characters-per-token heuristic, not a real BPE tokenizer. Real token counts are model-specific. They can differ from this estimate by a meaningful margin, especially on non-English text or heavy punctuation. The exact character count shown alongside every estimate is the one number here that isn't an estimate.

Should I chunk before or after extraction?

After, always. Chunking operates on whatever text extraction handed it. No chunking strategy recovers a reading order that extraction already destroyed. Paste your own extracted text above, or load the naive-PDF-text-dump sample. You'll see the same source content chunk visibly worse once extraction quality drops.

Why does a page number or header show up in every chunk?

A naive text extraction keeps whatever a page's layout put on it as ordinary lines of text, indistinguishable from content. That includes a running header, a footer, or a bare page number. Any chunker packs those lines in wherever they fall. So a repeated header can end up duplicated across dozens of chunks. The extraction-damage scan above flags repeated lines and bare page-number lines specifically, so you can catch this before it reaches your vector store.

What's the actual difference between the three strategies?

Fixed-size chunking cuts a hard window of N estimated tokens with some overlap, and ignores every boundary. Recursive chunking splits on paragraph, then line, then sentence, then word. It greedily packs units up to the target and only falls back to a smaller unit when the current one overflows. Structure-aware chunking sections the document on detected headings first. It then recursively splits any oversized section, and merges tiny adjacent sections back up toward the target.

See how your text will actually chunk.

Paste text you've already extracted, or load a sample, and watch chunk boundaries, token estimates, and extraction damage happen on real content. It runs entirely in your browser. Nothing is uploaded.

Paste text you've already extracted, or load a sample below, to see how it chunks. It runs entirely in your browser. Nothing is uploaded.

Extracted text

or load a local .txt/.md file

This is the text your extraction call already returns — the exact request:

curl

curl -X POST https://api.txtfetch.com/v1/extract \
  -H "Authorization: Bearer $TXTFETCH_KEY" \
  -F file=@document.pdf

Python

import os
import requests

with open("document.pdf", "rb") as f:
    r = requests.post(
        "https://api.txtfetch.com/v1/extract",
        headers={"Authorization": f"Bearer {os.environ['TXTFETCH_KEY']}"},
        files={"file": f},
    )

print(r.json()["extracted_text"])

JavaScript

import { readFile } from "node:fs/promises";

const file = new Blob([await readFile("document.pdf")]);
const form = new FormData();
form.append("file", file, "document.pdf");

const res = await fetch("https://api.txtfetch.com/v1/extract", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.TXTFETCH_KEY}` },
  body: form,
});

const { extracted_text } = await res.json();
console.log(extracted_text);

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"mime/multipart"
	"net/http"
	"os"
)

type extractResponse struct {
	Status        string `json:"status"`
	ExtractedText string `json:"extracted_text"`
}

func main() {
	f, err := os.Open("document.pdf")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	var body bytes.Buffer
	writer := multipart.NewWriter(&body)
	part, err := writer.CreateFormFile("file", "document.pdf")
	if err != nil {
		panic(err)
	}
	if _, err := io.Copy(part, f); err != nil {
		panic(err)
	}
	writer.Close()

	req, err := http.NewRequest("POST", "https://api.txtfetch.com/v1/extract", &body)
	if err != nil {
		panic(err)
	}
	req.Header.Set("Authorization", "Bearer "+os.Getenv("TXTFETCH_KEY"))
	req.Header.Set("Content-Type", writer.FormDataContentType())

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	var result extractResponse
	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
		panic(err)
	}
	fmt.Println(result.ExtractedText)
}

Chunking options

Strategy

Target size (est. tokens)

Overlap %

Embedding limit (tokens)

what-this-does

Chunking strategies for RAG argues that extraction quality bounds chunk quality. No chunker recovers a reading order that a mangled extraction already destroyed. This tool turns that argument into something you can prove on your own text. Paste it, pick a strategy, and see exactly where the boundaries land. It shows how many estimated tokens each chunk costs, and which specific extraction-damage signals are wrecking those boundaries. Nothing you paste or load ever leaves your browser. New to chunking or tokens as concepts? The glossary defines both plainly.

the-three-strategies

Fixed size. A hard window of N estimated tokens with some percentage overlap, cut without regard for sentence or paragraph boundaries. It gives simple, predictable chunk counts. It's also the strategy most likely to cut a chunk off mid-sentence. That's deliberate, so you can see the cost of ignoring structure.
Recursive. Splits on paragraph, then line, then sentence, then word. It greedily packs units up to the target size. It only falls back to a smaller unit when the current one is still too big. A hard character cut only happens when a single word overflows the target on its own.
Structure-aware. Detects heading-ish lines: markdown headings, short ALL-CAPS lines, numbered clauses like 4.2 Termination or Chapter N. It sections the document on them. It recursively splits any section that's still oversized. It merges tiny adjacent sections back up toward the target, so you don't get one chunk per one-line heading.

All three are implemented exactly as described in the chunking strategies guide. The tool and the writing can't contradict each other, because they share the same logic.

overlap-and-why

Overlap re-includes a slice of the previous chunk's trailing content at the start of the next chunk. That way, a sentence or idea that straddles a boundary still appears in full in at least one chunk. It isn't split with no full copy anywhere. Every chunk below shows its overlap. It's shown both as a highlighted span in the chunk text and as an exact character count. That way you can see precisely what's duplicated and how much. Too little overlap risks losing context at a boundary. Too much means embedding and storing the same content repeatedly, for no retrieval benefit past a certain point.

token-estimates-are-estimates

Every token count on this page is an estimate, not a real tokenizer result. The heuristic counts whitespace-delimited words. It splits each into its letter/digit run(s) and punctuation run(s). It charges roughly 4 characters per token, with a floor of one token per run. It is not a BPE tokenizer. Real token counts are model-specific. The same text can tokenize to meaningfully different counts across embedding models. The exact character count shown next to every estimate is the one number on this page that isn't an estimate. Treat the token figure as a sizing guide, not a ground truth.

extraction-damage-signals

Every finding below is a signal, not proof. A structural scan over plain text can be wrong about intent, so read the sample it shows before acting on it. Here's what each one does to chunking, and where it typically gets fixed. Each links to its own page under fixes for broken extracted text, with the real cause and an honest DIY remedy:

Signal	What it does to chunking	Where it's fixed
Hyphenated line breaks	A chunk boundary can land right at the hyphen, splitting one word into two fragments.	Fix hyphenated line breaks
Hard-wrapped lines	Chunking packs by line instead of by sentence or paragraph, ending chunks mid-thought.	Fix hard-wrapped lines
Repeated header/footer lines	Boilerplate gets pulled into every chunk that lines up with a page boundary.	Fix repeated headers and footers
Page-number lines	A bare number can get packed as content, or misread as a heading in structure mode.	Fix bare page-number lines
Mojibake	Garbled characters break sentence- and paragraph-boundary detection near them.	Fix mojibake and wrong-encoding text
Replacement characters	Marks a spot the original extraction couldn't decode at all — that text is already lost.	Fix mojibake and wrong-encoding text
Unexpanded ligatures	Reads as one unrecognized character to word- and token-boundary logic.	Fix ligatures and smart punctuation
Missing spaces between words	Two fused words are tokenized and chunked as a single unrecognizable unit.	Fix missing spaces between words
Collapsed long lines	Usually two interleaved columns read as one — any chunker packs both in the wrong order.	Fix columns out of order
Space-run table columns	Table rows collapse into plain text with no structural marker; column order scrambles.	Fix tables losing structure
No paragraph breaks	Recursive and structure-aware chunking both degrade to splitting on sentences or lines.	Fix missing paragraph breaks

The tool's headline verdict reports on two separate axes, and it always ranks them the same way: extraction damage outranks chunk sizing. A chunk running past your selected embedding limit is a sizing problem, one slider away from fixed. So it reads chunk sizing needs work, never "fix extraction first." There is nothing wrong with that document's extraction in that case. Damage signals, by contrast, are not fixable at chunk time at all. So any of them present reads fix extraction first, and the summary names which axis drove the call. Text carrying damage signals is never reported as clean.

same-document-two-extractions

The two "load sample" buttons above are the same source page, extracted two different ways. One is a clean extraction with real headings, paragraphs, and a small table. The other is a naive PDF text dump of the exact same content. It carries hyphenated line breaks, a running header, page numbers, mojibake, and a collapsed table. Load each one and compare: same words, differently extracted, visibly worse chunks. That gap is the whole argument for extracting well in the first place, made on identical content instead of asserted in prose.

what-to-do-next

Once your own extracted text chunks the way you expect, see the extraction call that produced it. It's the same request shown in the tool above: POST /v1/extract, a file or a URL. Back comes extracted_text. Feed that straight into whichever chunker you use downstream. See the chunk → embed → index recipe for the LangChain and LlamaIndex versions of the same pipeline.

faq

What chunk size should I use for RAG?: Recursive chunking with a target of 300–500 estimated tokens and 10–15% overlap is a reasonable default for most prose-heavy documents. It's the same default the chunking-strategies guide recommends. Reach for structure-aware chunking when the source has reliable headings, like technical docs or contracts with numbered clauses. Use fixed-size chunking only when you need predictable chunk counts more than clean boundaries.
How much overlap should chunks have?: 10–15% is a reasonable starting point. That's enough for a sentence straddling a boundary to still appear in full in at least one chunk. It isn't so much that you pay to embed and store the same text repeatedly. Overlap matters more for fixed-size chunking, which ignores sentence boundaries entirely. Recursive and structure-aware chunking mostly avoid cutting mid-sentence in the first place, so overlap matters less for them.
Why are my chunks cut off mid-sentence?: Either you're using fixed-size chunking, which cuts at a hard character window regardless of what it lands on. Or a single sentence in your source text is longer than your target chunk size. Recursive and structure-aware chunking both fall back to a word-level split when even one sentence alone overflows the target. There's no smaller boundary above a hard character cut. This tool flags every chunk that starts or ends mid-sentence, so you can see exactly where and why.
Does this tool upload my text anywhere?: No. Chunking and the extraction-quality scan both run entirely in your browser. Nothing you paste, load, or drop is sent over the network. The only thing that leaves your browser is the JSON report, if you choose to copy it.
How many tokens is my document, really?: This tool can only estimate. It counts letter/digit runs and punctuation runs and applies a documented ~4-characters-per-token heuristic, not a real BPE tokenizer. Real token counts are model-specific. They can differ from this estimate by a meaningful margin, especially on non-English text or heavy punctuation. The exact character count shown alongside every estimate is the one number here that isn't an estimate.
Should I chunk before or after extraction?: After, always. Chunking operates on whatever text extraction handed it. No chunking strategy recovers a reading order that extraction already destroyed. Paste your own extracted text above, or load the naive-PDF-text-dump sample. You'll see the same source content chunk visibly worse once extraction quality drops.
Why does a page number or header show up in every chunk?: A naive text extraction keeps whatever a page's layout put on it as ordinary lines of text, indistinguishable from content. That includes a running header, a footer, or a bare page number. Any chunker packs those lines in wherever they fall. So a repeated header can end up duplicated across dozens of chunks. The extraction-damage scan above flags repeated lines and bare page-number lines specifically, so you can catch this before it reaches your vector store.
What's the actual difference between the three strategies?: Fixed-size chunking cuts a hard window of N estimated tokens with some overlap, and ignores every boundary. Recursive chunking splits on paragraph, then line, then sentence, then word. It greedily packs units up to the target and only falls back to a smaller unit when the current one overflows. Structure-aware chunking sections the document on detected headings first. It then recursively splits any oversized section, and merges tiny adjacent sections back up toward the target.

That was one file. The API does the queue.

This page read your text on your own machine. The API reads a folder of them.

Read the quickstart →

Get an API key →