Can mojibake always be fixed after the fact?

Yes, as long as the original bytes are still available. Re-encoding the wrongly-decoded string back to bytes and re-decoding with the correct encoding recovers the original text, because nothing was actually lost, only misread.

How is mojibake different from a replacement character?

Mojibake is a successful decode using the wrong encoding, so it's reversible, because the right bytes are still there. A replacement character marks a decode that failed outright; there's no valid character behind it to recover.

Mojibake

Text decoded with the wrong character encoding: the right bytes, read by the wrong rulebook. Fully reversible by re-decoding correctly.

definition

Mojibake is the visible result of decoding a byte sequence with the wrong character encoding. It produces a string of technically valid but semantically wrong characters in place of the original text.

Also called: garbled text, encoding garbage

in-plain-terms

The classic case is UTF-8 bytes read as Windows-1252. A right single quote in a word like "won't" is three UTF-8 bytes. Reinterpreting them one byte at a time under the wrong encoding turns one character into three wrong ones. The bytes never changed. Only the rule used to turn them back into characters did.

Because the original bytes are intact, mojibake is reversible. Re-encode the wrongly-decoded string back to bytes, then decode those bytes with the correct encoding, and the original text returns. This is different from a replacement character, which marks a byte sequence that couldn't be decoded at all and has no bytes left to recover.

example

It wonâ€™t behave the way you expect

Classic UTF-8-decoded-as-Windows-1252 mojibake. The right single quote in "won't" is three UTF-8 bytes, misread one byte at a time.

why-it-matters

Mojibake breaks sentence- and word-boundary detection right where it occurs. A chunker draws boundaries blind through the damage, and embeddings computed over the garbled tokens pollute the vector space around genuinely related content.
Confusing mojibake with a replacement character wastes effort. Mojibake is a re-decode away from full recovery; a replacement character is not.

how-to-check

Paste a sample of the suspect text into the chunk previewer. Mojibake surfaces as a named, counted signal. Scan for mojibake

related-terms

faq

Can mojibake always be fixed after the fact?: Yes, as long as the original bytes are still available. Re-encoding the wrongly-decoded string back to bytes and re-decoding with the correct encoding recovers the original text, because nothing was actually lost, only misread.
How is mojibake different from a replacement character?: Mojibake is a successful decode using the wrong encoding, so it's reversible, because the right bytes are still there. A replacement character marks a decode that failed outright; there's no valid character behind it to recover.

related-reading

Characters & encoding

See the term in real output.

Drop a file into the free reader and watch it happen.

Open the file reader →

Get an API key →