How accurate is it? Here are the numbers — and how we got them.
Most vendors quote a self-reported figure and move on. Here's our full methodology, our per-category results, and where the numbers should and shouldn't be trusted.
aggregate results
The real extraction engine, run against a committed, human-checked corpus via dependency-injected mock Tika/mock VLM fixtures — no JVM, no network calls, fully reproducible. All four tiers, whole corpus, 10 documents (premium-vlm ran on the 3 hardest documents only — see the quality climb below).
| Tier | Docs | Text similarity | Table cell-F1 | Table structure | Reading order | Structure valid |
|---|---|---|---|---|---|---|
| baseline-text (Tika plain text, E1) | 10 | 92.0% | – | – | – | – |
| markdown (structured, standard) | 10 | 91.7% | 57.1% | 67.4% | 75.2% | 100.0% |
| json (element-JSON, standard) | 10 | 91.2% | 57.1% | 67.4% | 81.7% | 100.0% |
| premium-vlm (VLM tier) | 3 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
the quality climb
Standard vs premium, on the hard PDFs.
Table fidelity and reading order on the documents where Tika's structural recovery is weakest — multi-column layouts, scanned pages, wide tables. This is the clearest view of what the premium VLM tier buys: cell-F1 climbs from 0.0% to 100.0% on every hard-PDF category.
| Category | Tier | Table cell-F1 | Table structure | Reading order |
|---|---|---|---|---|
| pdf-complex | json (element-JSON, standard) | 0.0% | 0.0% | 50.0% |
| pdf-complex | premium-vlm (VLM tier) | 100.0% | 100.0% | 100.0% |
| pdf-scanned | json (element-JSON, standard) | 0.0% | 0.0% | 0.0% |
| pdf-scanned | premium-vlm (VLM tier) | 100.0% | 100.0% | 100.0% |
| table-heavy | json (element-JSON, standard) | 0.0% | 72.0% | 66.7% |
| table-heavy | premium-vlm (VLM tier) | 100.0% | 100.0% | 100.0% |
full per-category breakdown
Show all 10 categories, all tiers
| Category | Tier | Docs | Text similarity | Table cell-F1 | Reading order |
|---|---|---|---|---|---|
| email-eml | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – |
| email-eml | markdown (structured, standard) | 1 | 98.7% | 100.0% | 75.0% |
| email-eml | json (element-JSON, standard) | 1 | 100.0% | 100.0% | 100.0% |
| email-msg | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – |
| email-msg | markdown (structured, standard) | 1 | 98.8% | – | 75.0% |
| email-msg | json (element-JSON, standard) | 1 | 100.0% | – | 100.0% |
| html | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – |
| html | markdown (structured, standard) | 1 | 100.0% | 100.0% | 100.0% |
| html | json (element-JSON, standard) | 1 | 100.0% | 100.0% | 100.0% |
| office-docx | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – |
| office-docx | markdown (structured, standard) | 1 | 100.0% | 100.0% | 85.7% |
| office-docx | json (element-JSON, standard) | 1 | 92.3% | 100.0% | 100.0% |
| office-pptx | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – |
| office-pptx | markdown (structured, standard) | 1 | 100.0% | – | 100.0% |
| office-pptx | json (element-JSON, standard) | 1 | 100.0% | – | 100.0% |
| office-xlsx | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – |
| office-xlsx | markdown (structured, standard) | 1 | 100.0% | 100.0% | 100.0% |
| office-xlsx | json (element-JSON, standard) | 1 | 100.0% | 100.0% | 100.0% |
| pdf-complex | baseline-text (Tika plain text, E1) | 1 | 47.9% | – | – |
| pdf-complex | markdown (structured, standard) | 1 | 47.9% | 0.0% | 50.0% |
| pdf-complex | json (element-JSON, standard) | 1 | 47.9% | 0.0% | 50.0% |
| pdf-complex | premium-vlm (VLM tier) | 1 | 100.0% | 100.0% | 100.0% |
| pdf-scanned | baseline-text (Tika plain text, E1) | 1 | 84.6% | – | – |
| pdf-scanned | markdown (structured, standard) | 1 | 84.6% | 0.0% | 0.0% |
| pdf-scanned | json (element-JSON, standard) | 1 | 84.6% | 0.0% | 0.0% |
| pdf-scanned | premium-vlm (VLM tier) | 1 | 100.0% | 100.0% | 100.0% |
| pdf-simple | baseline-text (Tika plain text, E1) | 1 | 100.0% | – | – |
| pdf-simple | markdown (structured, standard) | 1 | 100.0% | – | 100.0% |
| pdf-simple | json (element-JSON, standard) | 1 | 100.0% | – | 100.0% |
| table-heavy | baseline-text (Tika plain text, E1) | 1 | 87.0% | – | – |
| table-heavy | markdown (structured, standard) | 1 | 87.0% | 0.0% | 66.7% |
| table-heavy | json (element-JSON, standard) | 1 | 87.0% | 0.0% | 66.7% |
| table-heavy | premium-vlm (VLM tier) | 1 | 100.0% | 100.0% | 100.0% |
methodology
This harness runs the real extraction engine (extract.js, render.js, premium.js) against a committed corpus, with a dependency-injected mock Tika (returns each corpus entry's recorded tika.xhtml/tika.txt) and a mock VLM (returns each entry's recorded vlm.md) — the same dependency-injection seam api/extract/src's own unit tests use. No JVM, no network calls; this runs on node:22-alpine with no npm install. Each corpus document carries a human-checked expected.json — the ideal reading order, text, and table structure, not a copy of what Tika produces. On easy formats (Office, HTML, email) Tika's own structure already matches the ideal closely. On hard PDFs (multi-column layouts, scanned/image-only pages, wide tables) the recorded tika.xhtml/tika.txt faithfully encode Tika's real failure modes — column-scrambled reading order, flattened or partially-recovered tables — while the recorded vlm.md encodes the correction a vision-capable model makes from the same page. The gap between the standard tiers and the premium tier on those documents is the quality climb this report exists to measure.
metric definitions
- Text similarity
- Normalized Levenshtein edit similarity between each tier's text projection and expected.text.
- Table cell-F1
- Table cells aligned by (row, col) position; a cell is "correct" when its normalized text matches the ideal. Headline table fidelity metric.
- Table structure ("TEDS-style")
- A lightweight cell-grid score: rewards matching (rows, cols) dimensions and matching cell positions, independent of text. This is honestly not the full academic tree-edit-distance TEDS metric (which diffs the underlying HTML/tree structure) — it's a cheap, zero-dependency stand-in for the same intuition.
- Reading order
- LCS of normalized block-text fingerprint sequences divided by the expected block count; penalizes both dropped and reordered blocks (the multi-column-PDF failure mode).
- Structure validity
- markdown: lightweight GFM well-formedness (balanced fences, consistent table column counts, no unclosed inline <table>). json: the shared validateElements from render-json.js (also used by api/extract/src's own tests).
Source of truth: api/extract/bench/REPORT.md, generated by `node api/extract/bench/run.js` (regenerate with `make bench`). This page hand-mirrors that committed artifact; it is not computed at request time.
caveats
- This corpus is small and synthetic (recorded fixtures standing in for real Tika/VLM responses) — it demonstrates the shape of the quality climb and guards against regression, it is not a claim of population-level accuracy.
- "TEDS-style" is a cell-grid F1/structural score, not the full academic TEDS metric — see Methodology above.
- Competitor figures are vendors' own published numbers on their own evaluation sets, gathered independently of this corpus — never apples-to-apples with the tables above. Several carry an explicit dispute or self-consistency caveat; read the per-vendor notes before citing any of them externally.
competitor context
Other vendors' own published numbers, on their own evaluation sets — NOT measured on our corpus, NOT apples-to-apples with the tables above. Read every note below before citing any of these figures externally.
These are OTHER VENDORS' OWN PUBLISHED NUMBERS on THEIR OWN evaluation sets and metric definitions — never measured on this repo's corpus, never independently reproduced by us. They are included for directional market context only, not as an apples-to-apples comparison against the tiers above. Every entry is flagged directional:true; most are also selfReported:true (the vendor's own blog/docs/marketing page). Where an entry notes a dispute or integrity concern, treat the number as contested, not authoritative.
LlamaParse (LlamaIndex)
Overall parse accuracy — "ParseBench": 84.9%directionalself-reported
Dataset: LlamaIndex's own ~2,000-page private ParseBench set (LlamaParse Agentic mode)
Source: LlamaIndex blog — "ParseBench" (accessed 2026-07)
Table TEDS — OmniDocBench (standardized dataset): 0.63 / 0.38 (two reported variants)directionalself-reported
Dataset: OmniDocBench (public/standardized)
Source: LlamaIndex blog — OmniDocBench post (accessed 2026-07)
Vendor-published on a standardized benchmark, but the post itself argues the metric undersells LlamaParse — read the scores alongside that framing.
Document QA accuracy — RealDoc-Bench (third party): 92.1%directional
Dataset: Extend.ai's RealDoc-Bench
Source: Extend.ai — RealDoc-Bench (accessed 2026-07)
Independent of LlamaIndex, but Extend is itself a competing document-AI vendor and ranks its own product (95.7%) above LlamaParse and Reducto (91.1%) on this set.
Unstructured.io
Table extraction score — "SCORE": 0.844 (blog post) / 0.880 (live benchmarks page, same claim)directionalself-reported
Dataset: Unstructured's own ~1,000-page private set
Source: Unstructured.io blog — SCORE benchmark (accessed 2026-07)
The two self-published figures for the same claim disagree — cite with caution.
Table TEDS — DP-Bench (standardized dataset): 65.56 (ranked 6th of 7 tools)directional
Dataset: Upstage DP-Bench (public)
Source: Upstage — DP-Bench paper (accessed 2026-07)
Azure AI Document Intelligence
Microsoft does not publish a fixed vendor accuracy percentage for Document Intelligence itself; every figure here is third-party, and third-party figures scatter roughly 65-89% depending on dataset.
Table TEDS — DP-Bench (standardized dataset): 77.85directional
Dataset: Upstage DP-Bench (public)
Source: Upstage — DP-Bench paper (accessed 2026-07)
Document QA accuracy — RealDoc-Bench (third party): 88.8%directional
Dataset: Extend.ai's RealDoc-Bench
Source: Extend.ai — RealDoc-Bench (accessed 2026-07)
AWS Textract
AWS does not publish a fixed accuracy percentage for Textract itself.
Table TEDS — third-party comparison: 80.75directional
Dataset: Tensorlake's published comparison set
Source: Tensorlake blog — parser comparison (accessed 2026-07)
Tensorlake is itself a competing document-parsing vendor.
Text-similarity accuracy — independent study: 99.3% (SBERT similarity, n=300 docs)directional
Dataset: AIMultiple's independent research sample
Source: AIMultiple Research — OCR accuracy comparison (accessed 2026-07)
Mindee
Field-level accuracy (vendor claim): >95% for most fieldsdirectionalself-reported
Dataset: Mindee's own private/rolling evaluation set (receipts/invoices)
Source: Mindee — product pages (accessed 2026-07)
Vague vendor marketing claim, not a fixed benchmark with a published dataset or date.
Reducto
Table extraction — "RD-TableBench": 90.2% (vs Azure DI 82.7%, Textract 80.9%, LlamaParse 74.6%, Unstructured 60.2%)directionalself-reported
Dataset: Reducto's own ~1,000-table private set, custom similarity metric (labelled "TEDS" but not the standard TEDS formula)
Source: Reducto blog — RD-TableBench (accessed 2026-07)
DISPUTED: an independent audit found a number of RD-TableBench's ground-truth files byte-identical to Reducto's own output, and errors shared with Reducto's own extraction — a material integrity concern. Treat this comparison as contested, not authoritative.
Dispute: Runpulse (Pulse AI) — RD-TableBench audit (accessed 2026-07)
Chunkr
No public vendor-vs-vendor accuracy benchmark found for Chunkr as of this writing — its published benchmark posts compare LLM/VLM choices inside its own pipeline on private eval sets, not against other document-parsing products. We deliberately cite no number here rather than infer one.
Nanonets
Overall score — OCR/parsing leaderboard: 85.9 (self-reported #1 of 29 models)directionalself-reported
Dataset: OlmOCR-bench / OmniDocBench / IDP Core (standardized public datasets, self-run and self-scored)
Source: Nanonets — public benchmarking page (accessed 2026-07)
The underlying datasets are standardized, but the ranking/run was performed and reported by Nanonets itself.
Mistral OCR
Overall accuracy / table accuracy — launch benchmark: 94.89% overall, 96.12% tables (vs GPT-4o 89.77%, Azure OCR 89.52%, Google Doc AI 83.42%)directionalself-reported
Dataset: Mistral's own private text-only evaluation set
Source: Mistral AI — "Mistral OCR" launch post (accessed 2026-07)
Contradicting third-party comparison: Gemini 80.1% vs Mistral OCR 45.3% on formsdirectional
Dataset: Reducto's own private RD-FormsBench
Source: Reducto blog — LVM OCR accuracy comparison (accessed 2026-07)
Reducto is a competing vendor with its own commercial interest in this comparison; the two vendors' benchmarks rank the field in opposite orders.
Stop parsing. Start shipping.
Create an account and get an API key in minutes — the free Hobby plan needs no card.
Get started →