> Source: https://txtfetch.com/benchmarks > Plain-text twin — every page on txtfetch.com has one. https://txtfetch.com/text --- # How accurate is it? Here are the numbers, and how we got them. Most vendors quote a self-reported figure and move on. Here's our full methodology, our per-category results, and where the numbers should and shouldn't be trusted. These are scores. Want the actual extracted text these numbers come from? [See it, side by side, at /diff →](https://txtfetch.com/diff) aggregate results This benchmark runs the real extraction engine against a committed, human-checked corpus, using dependency-injected mock Tika and mock VLM fixtures. It needs no JVM and no network calls, and it is fully reproducible. All four tiers, whole corpus, 10 documents (premium-vlm ran on the 3 hardest documents only; see the quality climb below). | Tier | Docs | Text similarity | Table cell-F1 | Table structure | Reading order | Structure valid | | --- | --- | --- | --- | --- | --- | --- | | baseline-text (Tika plain text, E1) | 10 | 92.0% | – | – | – | – | | markdown (structured, standard) | 10 | 91.7% | 57.1% | 67.4% | 75.2% | 100.0% | | json (element-JSON, standard) | 10 | 91.2% | 57.1% | 67.4% | 81.7% | 100.0% | | premium-vlm (VLM tier) | 3 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | the quality climb ## Standard vs premium, on the hard PDFs. This section covers table fidelity and reading order on the hardest documents: multi-column layouts, scanned pages, wide tables. Tika's structural recovery is weakest here. This is the clearest view of what the premium VLM tier buys. On every hard-PDF category, cell-F1 climbs from 0.0% to 100.0%. | Category | Tier | Table cell-F1 | Table structure | Reading order | | --- | --- | --- | --- | --- | | pdf-complex | json (element-JSON, standard) | 0.0% | 0.0% | 50.0% | | pdf-complex | premium-vlm (VLM tier) | 100.0% | 100.0% | 100.0% | | pdf-scanned | json (element-JSON, standard) | 0.0% | 0.0% | 0.0% | | pdf-scanned | premium-vlm (VLM tier) | 100.0% | 100.0% | 100.0% | | table-heavy | json (element-JSON, standard) | 0.0% | 72.0% | 66.7% | | table-heavy | premium-vlm (VLM tier) | 100.0% | 100.0% | 100.0% | full per-category breakdown **Show all 10 categories, all tiers** | Category | Tier | Docs | Text similarity | Table cell-F1 | Reading order | | --- | --- | --- | --- | --- | --- | | email-eml | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – | | email-eml | markdown (structured, standard) | 1 | 98.7% | 100.0% | 75.0% | | email-eml | json (element-JSON, standard) | 1 | 100.0% | 100.0% | 100.0% | | email-msg | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – | | email-msg | markdown (structured, standard) | 1 | 98.8% | – | 75.0% | | email-msg | json (element-JSON, standard) | 1 | 100.0% | – | 100.0% | | html | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – | | html | markdown (structured, standard) | 1 | 100.0% | 100.0% | 100.0% | | html | json (element-JSON, standard) | 1 | 100.0% | 100.0% | 100.0% | | office-docx | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – | | office-docx | markdown (structured, standard) | 1 | 100.0% | 100.0% | 85.7% | | office-docx | json (element-JSON, standard) | 1 | 92.3% | 100.0% | 100.0% | | office-pptx | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – | | office-pptx | markdown (structured, standard) | 1 | 100.0% | – | 100.0% | | office-pptx | json (element-JSON, standard) | 1 | 100.0% | – | 100.0% | | office-xlsx | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – | | office-xlsx | markdown (structured, standard) | 1 | 100.0% | 100.0% | 100.0% | | office-xlsx | json (element-JSON, standard) | 1 | 100.0% | 100.0% | 100.0% | | pdf-complex | baseline-text (Tika plain text, E1) | 1 | 47.9% | – | – | | pdf-complex | markdown (structured, standard) | 1 | 47.9% | 0.0% | 50.0% | | pdf-complex | json (element-JSON, standard) | 1 | 47.9% | 0.0% | 50.0% | | pdf-complex | premium-vlm (VLM tier) | 1 | 100.0% | 100.0% | 100.0% | | pdf-scanned | baseline-text (Tika plain text, E1) | 1 | 84.6% | – | – | | pdf-scanned | markdown (structured, standard) | 1 | 84.6% | 0.0% | 0.0% | | pdf-scanned | json (element-JSON, standard) | 1 | 84.6% | 0.0% | 0.0% | | pdf-scanned | premium-vlm (VLM tier) | 1 | 100.0% | 100.0% | 100.0% | | pdf-simple | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – | | pdf-simple | markdown (structured, standard) | 1 | 100.0% | – | 100.0% | | pdf-simple | json (element-JSON, standard) | 1 | 100.0% | – | 100.0% | | table-heavy | baseline-text (Tika plain text, E1) | 1 | 87.0% | – | – | | table-heavy | markdown (structured, standard) | 1 | 87.0% | 0.0% | 66.7% | | table-heavy | json (element-JSON, standard) | 1 | 87.0% | 0.0% | 66.7% | | table-heavy | premium-vlm (VLM tier) | 1 | 100.0% | 100.0% | 100.0% | methodology This harness runs the real extraction engine (extract.js, render.js, premium.js) against a committed corpus. It uses a dependency-injected mock Tika and a mock VLM. The mock Tika returns each corpus entry's recorded tika.xhtml/tika.txt. The mock VLM returns each entry's recorded vlm.md. This is the same dependency-injection seam that api/extract/src's own unit tests use. The harness needs no JVM and no network calls. It runs on node:22-alpine with no npm install. Each corpus document carries a human-checked expected.json file. This file holds the ideal reading order, text, and table structure. It is not a copy of what Tika produces. On easy formats (Office, HTML, email) Tika's own structure already matches the ideal closely. On hard PDFs (multi-column layouts, scanned/image-only pages, wide tables) the recorded tika.xhtml/tika.txt faithfully encode Tika's real failure modes. These failure modes include column-scrambled reading order and flattened or partially-recovered tables. The recorded vlm.md encodes the correction a vision-capable model makes from the same page. The gap between the standard tiers and the premium tier on those documents is the quality climb. This report exists to measure that gap. metric definitions **Text similarity**: Normalized Levenshtein edit similarity between each tier's text projection and expected.text. **Table cell-F1**: Table cells are aligned by (row, col) position. A cell is "correct" when its normalized text matches the ideal. Headline table fidelity metric. **Table structure ("TEDS-style")**: A lightweight cell-grid score: rewards matching (rows, cols) dimensions and matching cell positions, independent of text. This is not the full academic tree-edit-distance TEDS metric. That metric diffs the underlying HTML/tree structure. This score is a cheap, zero-dependency stand-in for the same intuition. **Reading order**: LCS of normalized block-text fingerprint sequences divided by the expected block count. This penalizes both dropped and reordered blocks (the multi-column-PDF failure mode). **Structure validity**: markdown: lightweight GFM well-formedness (balanced fences, consistent table column counts, no unclosed inline ). json: the shared validateElements from render-json.js (also used by api/extract/src's own tests). Source of truth: api/extract/bench/REPORT.md, generated by \`node api/extract/bench/run.js\` (regenerate with \`make bench\`). This page hand-mirrors that committed artifact. It is not computed at request time. caveats - This corpus is small and synthetic. It uses recorded fixtures that stand in for real Tika/VLM responses. It demonstrates the shape of the quality climb and guards against regression. It is not a claim of population-level accuracy. - "TEDS-style" is a cell-grid F1/structural score, not the full academic TEDS metric. See Methodology above. - Competitor figures are vendors' own published numbers, on their own evaluation sets. Vendors gathered these numbers independently of this corpus. They are never apples-to-apples with the tables above. Several carry an explicit dispute or self-consistency caveat. Read the per-vendor notes before citing any of them externally. competitor context **Other vendors' own published numbers**, on their own evaluation sets. These are NOT measured on our corpus. They are NOT apples-to-apples with the tables above. Read every note below before citing any of these figures externally. These are OTHER VENDORS' OWN PUBLISHED NUMBERS, on THEIR OWN evaluation sets and metric definitions. We never measured them on this repo's corpus. We never independently reproduced them. They are included for directional market context only, not as an apples-to-apples comparison against the tiers above. Every entry is flagged directional:true. Most are also selfReported:true (the vendor's own blog/docs/marketing page). Where an entry notes a dispute or integrity concern, treat the number as contested, not authoritative. LlamaParse (LlamaIndex) - Overall parse accuracy — "ParseBench": 84.9% directional self-reported Dataset: LlamaIndex's own ~2,000-page private ParseBench set (LlamaParse Agentic mode) Source: [LlamaIndex blog — "ParseBench"](https://www.llamaindex.ai/blog/parsebench) (accessed 2026-07) - Table TEDS — OmniDocBench (standardized dataset): 0.63 / 0.38 (two reported variants) directional self-reported Dataset: OmniDocBench (public/standardized) Source: [LlamaIndex blog — OmniDocBench post](https://www.llamaindex.ai/blog) (accessed 2026-07) This figure is vendor-published, on a standardized benchmark. The post itself argues the metric undersells LlamaParse. Read the scores alongside that framing. - Document QA accuracy — RealDoc-Bench (third party): 92.1% directional Dataset: Extend.ai's RealDoc-Bench Source: [Extend.ai — RealDoc-Bench](https://www.extend.ai/blog) (accessed 2026-07) This figure is independent of LlamaIndex. Extend is itself a competing document-AI vendor. It ranks its own product (95.7%) above LlamaParse and Reducto (91.1%) on this set. Unstructured.io - Table extraction score — "SCORE": 0.844 (blog post) / 0.880 (live benchmarks page, same claim) directional self-reported Dataset: Unstructured's own ~1,000-page private set Source: [Unstructured.io blog — SCORE benchmark](https://unstructured.io/blog) (accessed 2026-07) The two self-published figures for the same claim disagree. Cite with caution. - Table TEDS — DP-Bench (standardized dataset): 65.56 (ranked 6th of 7 tools) directional Dataset: Upstage DP-Bench (public) Source: [Upstage — DP-Bench paper](https://arxiv.org/abs/2412.04824) (accessed 2026-07) Azure AI Document Intelligence Microsoft does not publish a fixed vendor accuracy percentage for Document Intelligence itself. Every figure here is third-party. Third-party figures scatter roughly 65-89% depending on dataset. - Table TEDS — DP-Bench (standardized dataset): 77.85 directional Dataset: Upstage DP-Bench (public) Source: [Upstage — DP-Bench paper](https://arxiv.org/abs/2412.04824) (accessed 2026-07) - Document QA accuracy — RealDoc-Bench (third party): 88.8% directional Dataset: Extend.ai's RealDoc-Bench Source: [Extend.ai — RealDoc-Bench](https://www.extend.ai/blog) (accessed 2026-07) AWS Textract AWS does not publish a fixed accuracy percentage for Textract itself. - Table TEDS — third-party comparison: 80.75 directional Dataset: Tensorlake's published comparison set Source: [Tensorlake blog — parser comparison](https://www.tensorlake.ai/blog) (accessed 2026-07) Tensorlake is itself a competing document-parsing vendor. - Text-similarity accuracy — independent study: 99.3% (SBERT similarity, n=300 docs) directional Dataset: AIMultiple's independent research sample Source: [AIMultiple Research — OCR accuracy comparison](https://research.aimultiple.com/ocr-accuracy/) (accessed 2026-07) Mindee - Field-level accuracy (vendor claim): \>95% for most fields directional self-reported Dataset: Mindee's own private/rolling evaluation set (receipts/invoices) Source: [Mindee — product pages](https://mindee.com) (accessed 2026-07) Vague vendor marketing claim, not a fixed benchmark with a published dataset or date. Reducto - Table extraction — "RD-TableBench": 90.2% (vs Azure DI 82.7%, Textract 80.9%, LlamaParse 74.6%, Unstructured 60.2%) directional self-reported Dataset: Reducto's own ~1,000-table private set, custom similarity metric (labelled "TEDS" but not the standard TEDS formula) Source: [Reducto blog — RD-TableBench](https://reducto.ai/blog/rd-tablebench) (accessed 2026-07) DISPUTED: an independent audit found a number of RD-TableBench's ground-truth files byte-identical to Reducto's own output. The audit also found errors shared with Reducto's own extraction. This is a material integrity concern. Treat this comparison as contested, not authoritative. Dispute: [Runpulse (Pulse AI) — RD-TableBench audit](https://runpulse.com/blog) (accessed 2026-07) Chunkr No public vendor-vs-vendor accuracy benchmark for Chunkr was found, as of this writing. Its published benchmark posts compare LLM/VLM choices inside its own pipeline, on private eval sets. They do not compare Chunkr against other document-parsing products. We deliberately cite no number here rather than infer one. Nanonets - Overall score — OCR/parsing leaderboard: 85.9 (self-reported #1 of 29 models) directional self-reported Dataset: OlmOCR-bench / OmniDocBench / IDP Core (standardized public datasets, self-run and self-scored) Source: [Nanonets — public benchmarking page](https://benchmarking.nanonets.com) (accessed 2026-07) The underlying datasets are standardized, but the ranking/run was performed and reported by Nanonets itself. Mistral OCR - Overall accuracy / table accuracy — launch benchmark: 94.89% overall, 96.12% tables (vs GPT-4o 89.77%, Azure OCR 89.52%, Google Doc AI 83.42%) directional self-reported Dataset: Mistral's own private text-only evaluation set Source: [Mistral AI — "Mistral OCR" launch post](https://mistral.ai/news/mistral-ocr) (accessed 2026-07) - Contradicting third-party comparison: Gemini 80.1% vs Mistral OCR 45.3% on forms directional Dataset: Reducto's own private RD-FormsBench Source: [Reducto blog — LVM OCR accuracy comparison](https://reducto.ai/blog/lvm-ocr-accuracy-mistral-gemini) (accessed 2026-07) Reducto is a competing vendor with its own commercial interest in this comparison; the two vendors' benchmarks rank the field in opposite orders. ## Stop parsing. Start shipping. Create an account and get an API key in minutes. The free Hobby plan needs no card. [Get started →](https://app.txtfetch.com/signup)