Quality checks for extracted text

The concession first

A 200 response is not a quality signal. None of the checks below prove a document's text is correct. They find the documents that are probably wrong instead. That is a different, cheaper problem, and it is the one you can solve at scale.

Layer 1 — automatic checks, no ground truth needed

These run on the extracted text alone. No human, and no reference copy of the document, is needed. Each one is a real check in scanExtractionQuality(), with the threshold it fires at.

Two more checks need the file itself, not just the returned text, so they sit outside scanExtractionQuality().

Every finding here links to the fix page for that symptom: why it happens, and how to repair it.

Layer 2 — sampling

Layer 1 does not catch everything. A wrong number, a mistranslated word, or a table read in the right shape but the wrong values all pass every automatic check. Someone still has to read a sample.

• Read every document a layer-1 check flagged. That is not optional sampling; it is the cheapest signal you have.
• Stratify the rest by format. A PDF pipeline and a DOCX pipeline fail in different ways, so one clean sample of PDFs says nothing about your DOCX documents.
• Stratify by source, too. Documents from one scanner, one customer, or one upstream system tend to share the same defect.
• Treat the sample as a practical check, not a statistical guarantee. A clean sample of 20 documents does not prove the other 199,980 are clean.

Layer 3 — a golden set and a regression run

Keep 20 to 50 documents with human-checked expected text: the correct words, the correct table cells, the correct reading order. Re-run them on every pipeline change, before it ships, not after.

Score each run with the same metrics /benchmarks already publishes, so you are not inventing a new definition of "correct":

• Text similarity
• Table cell-F1
• Table structure ("TEDS-style")
• Reading order
• Structure validity

See /benchmarks for what each metric means and how it is scored. See /diff for the worked example: raw extraction next to the corrected text, on real documents.

A parser that silently drops content is the defect this page exists to catch. The reverse risk is real too: text a document's author expected to stay hidden, coming back intact. See personal data in extracted text for where that happens and what to do about it. Hidden instructions are the same risk: untrusted documents covers them.

What to do with a flagged document

Read the fix

Each layer-1 signal above links to the /fixes page for that symptom: why it happens, and how to repair it.

Re-run at a higher quality tier

Pass ?quality=premium and the document routes through a vision-language model instead of the standard text pass.

Quarantine it

Route a flagged document to a review queue instead of the index. A document you know is suspect is safer held back than shipped.

A runnable layer-1 check

A small Python function with five of the layer-1 checks from the table above, at the same thresholds.

import re
from collections import Counter

def quality_report(text):
    """Layer-1 checks, same thresholds as the published table."""
    findings = {}

    hyphen_breaks = len(re.findall(r"[A-Za-z]-\n[a-z]", text))
    if hyphen_breaks >= 1:
        findings["hyphenated-linebreaks"] = hyphen_breaks

    missing_spaces = len(re.findall(r"[a-z][A-Z]", text)) + len(re.findall(r"\.[A-Za-z]", text))
    if missing_spaces >= 5:
        findings["missing-spaces"] = missing_spaces

    lines = [l.strip() for l in text.split("\n") if 3 <= len(l.strip()) <= 80]
    repeats = max(Counter(lines).values(), default=0)
    if repeats >= 3:
        findings["repeated-lines"] = repeats

    long_lines = sum(1 for l in text.split("\n") if len(l) > 300 and not re.search(r"[.!?]", l))
    if long_lines >= 1:
        findings["collapsed-long-lines"] = long_lines

    if len(text) > 2000 and "\n\n" not in text:
        findings["no-paragraph-breaks"] = 1

    return findings

Frequently asked questions

Does a 200 response mean the extracted text is correct?

No. It means the request succeeded, not that the text is right. A scanned page with no OCR, a scrambled table, or mojibake can all return a clean 200. The checks on this page find the documents that are probably wrong, which is a cheaper problem than proving every document is right.

Can these checks prove my extracted text is accurate?

No. They catch known damage patterns: mojibake, fused words, scrambled columns, and the rest of the table above. A document can pass every check and still contain a wrong number or a mistranslated word. Only a golden set with human-checked expected text measures accuracy. Even that measures it on the documents in the set, not on every document you process.

How many documents should a golden set hold?

20 to 50 is enough to catch a regression without becoming a maintenance burden. Pick documents that cover your real formats and your hardest cases, not a random sample. Re-run the set on every pipeline change, and treat a score drop like a failing test.

What do I do with a document a check flags?

Read the matching /fixes page first; it explains the likely cause. Then re-run the document at a higher quality tier, or route it to a review queue instead of your index. See "what to do with a flagged document" above.