My extracted text is full of ’ and “.

looks-like

It won’t behave the way you expect—and “smart quotes” turn into this.

Classic UTF-8-decoded-as-Windows-1252 mojibake. The right single quote in "won't" is three UTF-8 bytes. Reinterpreting them one byte at a time turns that one character into the three wrong ones you see. The closing quote at the end looks like only two. Its third byte, 0x9D, is undefined in Windows-1252 and lands on an invisible control character. That same 0x9D is what breaks the naive round-trip fix below.

why-it-happens

Mojibake happens when bytes are decoded with the wrong character encoding, almost always UTF-8 bytes read as Windows-1252 (or the reverse). A UTF-8-encoded right single quote is three bytes: E2 80 99. Reinterpreted one byte at a time as Windows-1252, it renders as ’: three separate "wrong" characters standing in for the one correct one.

It's a decoding bug, not a content problem. The original bytes are intact and correct. Only the encoding label applied when turning them into text is wrong. That's why re-decoding with the correct encoding, or round-tripping through the wrong one, recovers the original text exactly. No OCR or reconstruction is needed. The one wrinkle is mechanical rather than lossy. Windows-1252 has five undefined byte values, so a strict round trip needs to pass those through explicitly. The fix-it-yourself section below shows how.

The U+FFFD replacement character (�) is a different, worse failure. A decoder hit a byte sequence it couldn't map to any character in the encoding it assumed, and substituted the generic "unknown" glyph. Unlike mojibake, this is not reversible. The original byte value is gone, replaced permanently, so no re-decode recovers it.

confirm-it

• Paste the extracted text into the chunk previewer. Both mojibake and stray replacement characters are named, counted signals in its extraction-quality scan. Scan for mojibake and replacement characters

fix-it-yourself

pip install ftfy
python -c "import ftfy; print(ftfy.fix_text(open('extracted.txt').read(), uncurl_quotes=False))"

fixed = broken.encode("windows-1252").decode("utf-8")

def undo_cp1252_mojibake(s):
    out = bytearray()
    for ch in s:
        try:
            out += ch.encode("windows-1252")
        except UnicodeEncodeError:
            out.append(ord(ch))  # 0x81 0x8d 0x8f 0x90 0x9d — undefined in cp1252
    return out.decode("utf-8")

from charset_normalizer import from_bytes
result = from_bytes(raw_bytes).best()
text = str(result)

what-txtfetch-does

txtfetch's extraction runs through Apache Tika, which handles source-encoding detection as part of parsing rather than assuming UTF-8. The double-decode failure mode above is largely avoided for documents Tika parses directly.

That isn't a guarantee against every source. A document's text can already be mojibake'd before it ever reaches Tika. The damage, baked in by an earlier, unrelated conversion step, comes back exactly as damaged as the source. There's no way to distinguish "wrong on purpose" from "wrong by an upstream mistake" after the fact.

what-it-costs-you-downstream

Garbled runs break sentence- and paragraph-boundary detection right where they occur, so recursive and structure-aware chunking can't reliably find clean edges around them. The corrupted tokens themselves add embedding noise exactly where real, meaningful words used to be.

faq

Can I always fix mojibake after the fact?

Usually, yes. Mojibake preserves the original bytes, just decoded with the wrong encoding. So re-decoding correctly, or using a tool like ftfy, recovers the original text exactly. Mind one mechanical trap: the naive broken.encode("windows-1252") round trip raises UnicodeEncodeError whenever the damage includes one of cp1252's five undefined bytes. A mojibaked curly double quote always does. The fix-it-yourself section above has the version that handles it. A replacement character (�) is the genuinely unrecoverable case: the byte was lost during decoding, and no amount of re-decoding brings it back.

How do I tell mojibake apart from a replacement character?

Mojibake reads as a run of plausible-looking wrong letters, like ’ or é. A replacement character is the single glyph � standing alone, or the diamond-question-mark you sometimes see instead. The chunk previewer's quality scan reports them as two separate, named signals for exactly this reason.

Does txtfetch produce mojibake on documents it extracts?

Rarely, since Tika detects source encoding as part of parsing rather than assuming UTF-8. It can still happen if the mojibake was already present in the source document's text before txtfetch ever saw it. There's no way to tell 'broken on purpose' from 'broken upstream' from the bytes alone.

related-reading

• Garbled PDF text (a different cause) →
• Chunk previewer →
• PDF text-layer checker →