https://txtfetch.com/fixes/headers-and-footers-in-every-chunk/
The same header shows up at the top of every single chunk.
A running header or footer is drawn on every page as part of the layout. Extraction has no concept of "this line is chrome, not content", so it reads it as body text, every time, on every page.
looks-like
Confidential — Acme Corp Internal Use Only
...page content...
Confidential — Acme Corp Internal Use Only
3
...more content...The confidentiality stamp and the bare page number both extract as ordinary text, identical to the surrounding content, on every page.
why-it-happens
A running header or footer, such as a document title, a confidentiality stamp, or a page number, is drawn on every page. It's part of the page's own layout, at the same position each time. A layout-naive text extractor has no concept of "this line is chrome, not content". It reads every page's text top to bottom, including whatever repeats identically at the top or bottom, and folds it straight into the body.
A bare page number is a related but distinct case. "3" or "Page 3 of 42" on its own line is real, extractable text. It's just not content. In structure-aware chunking it can even get misread as a heading, since it's a short, isolated line much like a real one.
confirm-it
- Paste extracted text into the chunk previewer. A line repeating three or more times, or two or more bare page-number-shaped lines, both trip named signals. Scan for repeated lines and page numbers
fix-it-yourself
Strip lines that repeat across pages
A frequency filter over line-level text removes boilerplate that shows up far more often than any real sentence would. It does this without needing to know the document's layout in advance.
python
from collections import Counter
lines = text.split("\n")
counts = Counter(l.strip() for l in lines if 3 <= len(l.strip()) <= 80)
boilerplate = {l for l, c in counts.items() if c >= 3}
cleaned = "\n".join(l for l in lines if l.strip() not in boilerplate)Drop bare page-number lines with a regex
Matches "Page 3", "Page 3 of 42", a bare "3", or "- 3 -" on their own line. These are the common shapes a page-number footer takes once extracted as text.
python
import re
PAGE_NUM_RE = re.compile(r"^(?:page\s+)?\d{1,4}(?:\s*(?:of|/)\s*\d{1,4})?$|^-\s*\d{1,4}\s*-$", re.I | re.M)
cleaned = "\n".join(l for l in text.split("\n") if not PAGE_NUM_RE.match(l.strip()))what-txtfetch-does
txtfetch returns exactly what Tika reads from the document's content stream. It doesn't run a header/footer-stripping heuristic on the default text response. "Is this line chrome or content" is a document-layout judgment call. It's best made with the frequency-across-pages signal above, on your side, where you control how aggressive to be.
This is deliberately a caller-side post-processing step, not something the extraction API decides for you. It's the same reasoning /fixes/hard-wrapped-lines and /fixes/no-paragraph-breaks apply to their own post-processing concerns.
what-it-costs-you-downstream
A repeated header gets pulled into every chunk that lines up with a page boundary. This wastes embedding budget on the same few words, dozens of times over, instead of unique content. A bare page number can get packed into a chunk as if it carried meaning, or worse, mistaken for a section heading in structure-aware chunking.
faq
- Why doesn't txtfetch just strip repeated headers automatically?
- Deciding what's boilerplate versus real repeated content is a judgment call specific to your document, not a fact the extraction step can know for certain. A short line that repeats could be a header, or it could be a genuinely repeated piece of content. txtfetch returns exactly what Tika reads; stripping is a deliberate caller-side step.
- How many repeats does the chunk previewer need to flag a line as a header?
- Three or more identical occurrences for the repeated-line signal, or two or more bare page-number-shaped lines for the page-number signal. This is tuned so a coincidental one-off repeated phrase doesn't trigger a false positive.
- Can a page number get misread as a heading?
- Yes, specifically in structure-aware chunking. A bare short line like "3" resembles the shape of an all-caps or numbered-clause heading. Left unfiltered, it can seed a spurious section boundary.
related-reading
Fix the text you already have.
The free cleaner repairs this damage in your browser. Nothing leaves the page.
Clean up your text →