Is hard-wrapped text actually damaged, or just formatted differently?

The words themselves are intact. Nothing is lost, unlike mojibake or a missing ToUnicode CMap. The damage is structural: a chunker that reads line by line, instead of by paragraph, ends chunks wherever the print page ran out of width. It doesn't end them where the idea actually ended.

How do I reflow hard-wrapped text without merging real paragraph breaks?

Only join a line into the previous one when the previous line doesn't already end with sentence-ending punctuation. A real paragraph break (a blank line) is a different pattern entirely and should be left alone. See the fix-it-yourself snippet above.

Does txtfetch hard-wrap its output?

No. Tika's PDF and Office parsing generally reflows paragraphs rather than preserving literal print-line breaks. Hard-wrapping mostly shows up on plain-text or heavily fixed-width sources where there's no paragraph structure to reflow from in the first place.

Every line in my extracted text ends at the same width, mid-sentence.

Some extraction paths preserve every visual line break from the source layout literally. It's readable to a human, but each print-width line becomes its own unit to a chunker.

looks-like

This is a long paragraph that has been hard-wrapped by a
naive text extraction pipeline instead of being reflowed into
one continuous line the way a real paragraph normally would
be represented in clean extracted text.

Visually readable line by line. That's exactly why it's easy to miss until a chunker treats every line as its own unit.

why-it-happens

Some extraction paths preserve every visual line break from the source layout literally. Each line the page happened to wrap at roughly 60-80 characters becomes its own line in the output, with a real newline character. It isn't reflowed into the single logical paragraph a human reader experiences it as.

This is common output from plain-text renderings of fixed-width layouts. Certain OCR post-processing steps also emit one line per detected text line, as do PDF extractors running without a reflow or de-hyphenation pass.

It's visually readable to a human. The paragraph still makes sense line by line. That's exactly why it's easy to miss until a chunker treats every hard-wrapped line as its own unit.

confirm-it

Paste the text into the chunk previewer. Three or more lines that read like prose, don't end a sentence, and continue lowercase on the next line trip this signal. Scan for hard-wrapped lines

fix-it-yourself

Reflow with a single-newline-to-space join

Joins a line into the previous one only when the previous line doesn't end a sentence. Real paragraph breaks (blank lines) are left alone.

python

import re
def reflow(text):
    lines = text.split("\n")
    out = []
    for line in lines:
        if out and out[-1] and not re.search(r"[.!?:;]\s*$", out[-1]) and line and line[0].islower():
            out[-1] = out[-1] + " " + line
        else:
            out.append(line)
    return "\n".join(out)

Better, when blank lines already mark the paragraphs

If the source kept its blank lines between paragraphs, don't guess at sentence boundaries at all. Split on the blank lines first, then collapse every newline inside each block. The paragraph edges are explicit, so this can't merge two paragraphs the way a sentence-boundary heuristic can. That heuristic can misfire when a paragraph ends without punctuation, like a heading, a list item, or a table cell. Stdlib only.

python

import re

def reflow_paragraphs(text):
    blocks = re.split(r"\n\s*\n", text)
    return "\n\n".join(re.sub(r"\s*\n\s*", " ", b).strip() for b in blocks)

what-txtfetch-does

This is a caller-side post-processing concern, not something the extraction step decides on your behalf. txtfetch returns the text Tika produces, and Tika generally reflows PDF and Office paragraphs rather than preserving literal print-line breaks. So hard-wrapping shows up far less than in a raw fixed-width text dump or a naive OCR-to-text pipeline.

It does happen most often on plain-text or heavily fixed-width sources, where there's no paragraph model to reflow from in the first place. The fix above is intentionally something you own. "Is this a real line break or a print-width wrap" is a judgment call about the source document. It's not a fact the extraction step can always know for certain.

what-it-costs-you-downstream

Every hard-wrapped line reads as its own short unit to recursive and structure-aware chunking, which packs chunks by line instead of by sentence or paragraph. Chunks end mid-thought exactly where the printed page happened to run out of width, not where the idea actually ended.

faq

Is hard-wrapped text actually damaged, or just formatted differently?: The words themselves are intact. Nothing is lost, unlike mojibake or a missing ToUnicode CMap. The damage is structural: a chunker that reads line by line, instead of by paragraph, ends chunks wherever the print page ran out of width. It doesn't end them where the idea actually ended.
How do I reflow hard-wrapped text without merging real paragraph breaks?: Only join a line into the previous one when the previous line doesn't already end with sentence-ending punctuation. A real paragraph break (a blank line) is a different pattern entirely and should be left alone. See the fix-it-yourself snippet above.
Does txtfetch hard-wrap its output?: No. Tika's PDF and Office parsing generally reflows paragraphs rather than preserving literal print-line breaks. Hard-wrapping mostly shows up on plain-text or heavily fixed-width sources where there's no paragraph structure to reflow from in the first place.

related-reading

Layout & structure

Fix the text you already have.

The free cleaner repairs this damage in your browser. Nothing leaves the page.

Clean up your text →

Get an API key →