Is the hyphen in inter-\nnational a real hyphen from the source document?

No. It's a print-layout artifact from where the page happened to wrap the word across two lines. The source word is "international", with no hyphen at all. A naive extractor just preserves the visual line break literally.

Will the de-hyphenation regex ever join text it shouldn't?

It's written to only fire when the character right after the break is lowercase. So a genuine hyphenated compound followed by a new, capitalized sentence is left alone. It can still occasionally misfire on an intentional hyphenated word that happens to wrap at a line break, so spot-check the output.

Does txtfetch fix this automatically?

Mostly. Tika's PDF parsing reflows paragraphs rather than preserving literal print-line positions, so this shows up far less than in a raw layout-preserving extraction. It's a reflow heuristic, not a guarantee on every document, so the fix above is worth keeping as a backstop.

My extracted text still has inter-\nnational split across a line break.

A justified print layout breaks a long word at the end of a line and marks it with a hyphen. A naive extractor preserves that hyphen and the newline literally. The source word never had either.

looks-like

The report covers the inter-
national deployment process in detail.

"international" was never meant to contain a hyphen or a line break. It only exists because the printed page wrapped there.

why-it-happens

A justified or hyphenated print layout breaks a long word across the end of one visual line and the start of the next. It marks the break with a hyphen: "inter-" ends one line, "national" starts the next. A naive extractor preserves the source's original line breaks literally. So the hyphen and the newline both survive into the output exactly where the printed page happened to wrap the word.

This is a print-layout artifact, not a real hyphen. The source document's word "international" was never meant to contain either character. It exists purely because the page has a fixed width and the word didn't fit on one line.

confirm-it

Paste the text into the chunk previewer. It flags every letter-hyphen-newline-lowercase break specifically, with a live count. Scan for hyphenated line breaks

fix-it-yourself

Re-join with a de-hyphenation regex

Only joins when the character after the break is lowercase. A genuine end-of-sentence hyphen followed by a capitalized new sentence is left alone.

python

import re
text = re.sub(r"([A-Za-z])-\n([a-z])", r"\1\2", text)

Extract with reflow instead of raw line preservation

pdftotext without -layout already reflows most paragraphs. This avoids introducing the hyphenated break in the first place rather than fixing it after the fact.

bash

pdftotext input.pdf output.txt

what-txtfetch-does

Tika's PDF extraction reflows paragraph text rather than preserving the source's literal line-wrap positions. So hyphenated line-break artifacts show up substantially less than in a raw layout dump.

Reflow heuristics aren't perfect on every producer, particularly dense multi-column layouts. It's worth spot-checking on documents where retrieval quality matters, using the de-hyphenation regex above as a safety net.

what-it-costs-you-downstream

A chunk boundary can land exactly at the hyphen, turning one word into two separate chunk-boundary fragments. "inter-" ends up at the end of one chunk and "national" at the start of the next. Neither one embeds or matches a search for "international".

faq

Is the hyphen in inter-\nnational a real hyphen from the source document?: No. It's a print-layout artifact from where the page happened to wrap the word across two lines. The source word is "international", with no hyphen at all. A naive extractor just preserves the visual line break literally.
Will the de-hyphenation regex ever join text it shouldn't?: It's written to only fire when the character right after the break is lowercase. So a genuine hyphenated compound followed by a new, capitalized sentence is left alone. It can still occasionally misfire on an intentional hyphenated word that happens to wrap at a line break, so spot-check the output.
Does txtfetch fix this automatically?: Mostly. Tika's PDF parsing reflows paragraphs rather than preserving literal print-line positions, so this shows up far less than in a raw layout-preserving extraction. It's a reflow heuristic, not a guarantee on every document, so the fix above is worth keeping as a backstop.

related-reading

Characters & encoding

Fix the text you already have.

The free cleaner repairs this damage in your browser. Nothing leaves the page.

Clean up your text →

Get an API key →