> Source: https://txtfetch.com/fixes/ligatures-and-smart-punctuation > Plain-text twin — every page on txtfetch.com has one. https://txtfetch.com/text --- # My extracted text has fi and fl instead of plain letters. A ligature is one glyph standing in for a letter pair, encoded as its own character. Smart quotes and non-breaking spaces are similar, deliberate substitutions. They're correct in the source, but different from what a naive string match expects. looks-like ``` The office workflow relies on efficient filing — but ligature glyphs like fi and fl don't always expand back to plain letters. ``` office and workflow each carry a single ligature glyph (ffi, U+FB03; fl, U+FB02) where a search or tokenizer expects two plain letters. why-it-happens Typography renders certain letter pairs, fi, fl, ff, ffi, ffl, as one combined glyph for visual quality, particularly in serif fonts. The PDF's font can encode that combined shape as its own character code, distinct from separate "f" and "i" codes. So an extractor reading character codes literally gets back one ligature character (fi, U+FB01) rather than the two ASCII letters a search or tokenizer expects. Smart, or "curly", punctuation is a related but separate substitution. Word processors and typesetting systems replace straight quotes and hyphens with their typographic equivalents (', ', ", ", –, —) as you type. This is correct and intentional in the source, but a different codepoint than the plain ASCII a naive string match expects. Non-breaking spaces (U+00A0) and soft hyphens (U+00AD) are two more invisible-looking codepoints that behave differently from what they appear to be. A non-breaking space looks like a space but doesn't always split the same way under whitespace-based tokenization. A soft hyphen is invisible unless a line actually breaks there, but it survives into extracted text as a stray character regardless. confirm-it - Paste the text into the chunk previewer. Unexpanded ligature glyphs are a named, counted signal in its extraction-quality scan. [Scan for unexpanded ligatures](https://txtfetch.com/tools/chunk-preview) fix-it-yourself Expand ligatures with NFKC normalization NFKC (compatibility) normalization is the right tool for ligatures specifically: fi becomes fi, ffi becomes ffi. It also folds a non-breaking space to a plain space and an ellipsis character to three dots. What it does NOT touch is smart punctuation. Curly quotes and en/em dashes aren't compatibility characters, so they survive NFKC unchanged, and a soft hyphen does too. Those need the explicit map below. python ``` import unicodedata text = unicodedata.normalize("NFKC", text) ``` Fold smart punctuation with an explicit map This is the half NFKC leaves alone: curly quotes, en/em dashes, and the soft hyphen. A translate() map is better than chained replace() calls here, because it's one pass and the intent is legible. Add or drop a codepoint as your downstream matching requires. Keep the NBSP and soft-hyphen entries even if you also ran NFKC. An NBSP that survives silently breaks whitespace-based tokenizers. A soft hyphen is invisible, right up until it splits a word for you. python ``` PUNCT = { 0x2018: "'", 0x2019: "'", # ' ' curly single quotes 0x201c: '"', 0x201d: '"', # " " curly double quotes 0x2013: "-", 0x2014: "-", # – — en/em dash 0x00a0: " ", # non-breaking space 0x00ad: "", # soft hyphen — delete outright } text = text.translate(PUNCT) ``` what-txtfetch-does Apache Tika expands most common ligatures during PDF and Office text extraction as part of its normal glyph-to-Unicode mapping. So this shows up far less than in a raw PDF-library dump. It isn't guaranteed for every font and producer combination, though. Smart punctuation is passed through as-is, deliberately. Curly quotes and em dashes are the correct rendering of the source document, not an extraction defect, so txtfetch doesn't rewrite them. Fold them on your side if your downstream matching needs plain ASCII. Note that NFKC alone won't do it, since those characters aren't compatibility characters. The explicit translate() map above is what handles them. what-it-costs-you-downstream An unexpanded ligature reads as one unrecognized character to word- and token-boundary logic. "office" doesn't tokenize as, embed as, or lexically match "office". This quietly shifts both token counts and retrieval recall near every occurrence. faq **Should I normalize curly quotes to straight quotes before embedding?**: Only if your downstream matching (exact-string search, a keyword filter) needs plain ASCII. Curly quotes are the source document's correct, intentional typography, not a defect. Semantic embedding models generally handle both forms fine on their own. **Does NFKC normalization handle curly quotes and em dashes too?**: No, and this is the most common wrong assumption about NFKC. It expands ligatures (fi to fi), folds a non-breaking space to a plain space, and turns an ellipsis character into three dots. But curly quotes, en dashes, em dashes, and soft hyphens are not compatibility characters. NFKC leaves every one of them exactly as it found them. Use an explicit translate() map for those; there's one in the fix-it-yourself section above. **Does NFKC normalization ever change meaning, not just form?**: It can. NFKC folds some distinct-looking characters toward a shared compatibility form, which is what you want for ligatures. But it occasionally collapses characters that were meaningfully distinct in specialized text. Some mathematical notation is one example. East Asian full-width forms folding to their half-width equivalents are another. For ordinary English prose extraction it's a safe default. Spot-check it if your corpus is mathematical or CJK. **Why do only some fonts produce ligature characters in extracted text?**: It depends on whether the font encodes fi/fl/ff as their own glyph codes, or as separate letter codes rendered close together. The former is common in serif fonts optimized for print. Tika expands the common cases, but isn't guaranteed across every font and producer. related-reading - [Mojibake / wrong encoding →](https://txtfetch.com/fixes/mojibake-wrong-encoding) - [Chunk previewer →](https://txtfetch.com/tools/chunk-preview) Characters & encoding - [extracted text is full of mojibake or replacement characters →](https://txtfetch.com/fixes/mojibake-wrong-encoding) - [missing spaces between words in extracted text →](https://txtfetch.com/fixes/missing-spaces-between-words) - [hyphenated line breaks left in extracted text →](https://txtfetch.com/fixes/hyphenated-line-breaks) - [All fixes →](https://txtfetch.com/fixes) ## Fix the text you already have. The free cleaner repairs this damage in your browser. Nothing leaves the page. [Clean up your text →](https://txtfetch.com/tools/clean-extracted-text) [Get an API key →](https://app.txtfetch.com/signup)