Can I recover the text a replacement character replaced?

Generally no. It marks bytes that couldn't be decoded as any valid character under the encoding applied. The decode failed outright, rather than producing the wrong character, so there's no encoding fix that reconstructs what was there.

Should I just strip replacement characters from extracted text?

Only after understanding what caused them. A spike is a signal of real data loss at that point in the document, worth investigating before treating it as a cosmetic cleanup step.

Replacement character

Marks a byte sequence a decoder couldn't turn into any real character at all. Unlike mojibake, there's nothing left to recover.

definition

The replacement character (Unicode U+FFFD) is the placeholder a decoder substitutes for a byte sequence it cannot read. This happens when the bytes are not a valid character under the encoding in use.

Also called: U+FFFD, mojibake vs. replacement character

in-plain-terms

Sometimes a decoder hits bytes that don't form a valid character under the encoding it's applying. Truncated multi-byte UTF-8 sequences and genuinely corrupted data are two examples. When this happens, the decoder has no correct character to produce. Rather than crash or silently drop the bytes, most decoders substitute the replacement character and continue. The rest of the document still comes through.

This is a stronger failure than mojibake. Mojibake is the wrong character produced from a valid decode with the wrong encoding. It's reversible by re-decoding correctly, because the original bytes are recoverable from what came out. A replacement character means the decode failed outright. The bytes it stood in for are gone from the output, and there's no encoding fix that brings them back.

example

The scanner hit a byte sequence it could not decode: � and continued

A single replacement character marks exactly where decoding failed. Everything before and after it decoded fine.

why-it-matters

A spike in replacement characters is a strong signal of real, unrecoverable data loss at that point in the text. It's not a cosmetic glitch to clean up after the fact.
Confusing a replacement character with mojibake leads to the wrong fix. Re-decoding with a different encoding does nothing for a replacement character, because there's no valid decode to find.

how-to-check

Count replacement characters and their surrounding context in extracted text. A spike is a signal worth investigating, not something to filter out and ignore. Scan for replacement characters

related-terms

faq

Can I recover the text a replacement character replaced?: Generally no. It marks bytes that couldn't be decoded as any valid character under the encoding applied. The decode failed outright, rather than producing the wrong character, so there's no encoding fix that reconstructs what was there.
Should I just strip replacement characters from extracted text?: Only after understanding what caused them. A spike is a signal of real data loss at that point in the document, worth investigating before treating it as a cosmetic cleanup step.

related-reading

Characters & encoding

See the term in real output.

Drop a file into the free reader and watch it happen.

Open the file reader →

Get an API key →