Extract text from Japanese documents, kanji and kana intact.

日本語 · Han, Hiragana, Katakana

what-already-works

A born-digital Japanese file, a Word document or a modern PDF with a real text layer, needs no OCR at all. Tika reads its Unicode text layer directly, the same way it reads English. Kanji, hiragana, and katakana are each an ordinary Unicode code point, so the script isn't the hard part.

character-encodings

common-pitfalls

Every kanji and kana character turns into a run of unrelated symbols or question marks.

The file was Shift_JIS, EUC-JP, or ISO-2022-JP, and got read as UTF-8 or Latin-1 instead. This is mojibake from a legacy Japanese encoding, not lost data. Re-read the file with its real encoding. Fix garbled text from the wrong encoding →

A downstream word-count chunk size is wildly wrong for Japanese text.

Japanese writing has no spaces between words. A word-count chunker, built to split on spaces, sees the whole passage as one long 'word'. Chunk Japanese text by character or token count instead of by word count. What chunking actually does →

The same word appears in a document twice, once full-width and once half-width, and a search only finds one.

Japanese text mixes full-width (zenkaku) and half-width (hankaku) forms for numbers and Latin letters. These are different Unicode code points. Normalize full-width and half-width variants to one form before comparing or indexing the extracted text.

chunking-and-tokens

Japanese has no spaces between words, so a word-count chunker cannot split it at all. Each character also carries more meaning than a Latin letter does. The chunk previewer's ~4-characters-per-token estimate badly understates the real token count for Japanese. A real tokenizer usually spends closer to one token per character, not one per four.

ocr-status

A scanned or photographed Japanese document is a different problem than a born-digital one. The standard OCR tier ships English trained data plus script detection only (TESSDATA_LANGS=eng osd). Script detection can tell a page isn't Latin script. It cannot turn Japanese glyphs into Japanese text, since that needs Japanese-trained OCR data the standard tier doesn't ship. For a scanned Japanese document, request quality=premium, which routes a vision-language model instead of Tika. No accuracy figure is published for Japanese OCR under either tier. If scanned Japanese is a large part of your workload, contact us before you commit to a plan.

faq

Does txtfetch detect or label the document's language?

No. txtfetch extracts the text as written and hands it back as-is. It does not detect, tag, or translate a document's language. There is no language parameter to set.

Can txtfetch OCR a scanned Japanese document?

Not reliably with the standard tier, which ships English trained data plus script detection only (TESSDATA_LANGS=eng osd). Request quality=premium, which routes a vision-language model instead of Tika, for the documented route to reading a Japanese scan.

Why is my word-count chunk size wrong for Japanese text?

Japanese has no spaces between words, so a word-count chunker can't split it correctly. Chunk by character or token count instead, and treat the chunk previewer's character-based estimate as a rough floor, not an exact count.

Why does my Japanese text show unrelated symbols instead of kanji?

The source file likely used Shift_JIS, EUC-JP, or ISO-2022-JP, and something read it as UTF-8. Re-reading it with the correct encoding fixes the mojibake.

related-reading

• What comes back: text, Markdown, or JSON →
• Chunk previewer →
• Glossary: token →
• Contact →
• Get an API key →