PDF to Markdown Benchmarks
Benchmark methodology for evaluating PDF-to-Markdown tools on invoices, scanned pages, tables, and research papers.
This page defines the benchmark methodology before publishing scored vendor claims. It is intentionally conservative: do not treat unverified marketing claims as accuracy data.
First published run: the Japanese & Chinese OCR benchmark — measured character error rates and table-structure comparison vs Tesseract and plain text extraction, fully reproducible.
Document set
- One native invoice with line-item tables.
- One scanned invoice with low-resolution text.
- One two-column research paper.
- One table-heavy financial report.
- One legal contract with nested clauses and schedules.
Tools to evaluate
Adobe PDF Extract, Mathpix, LlamaParse, Unstructured, Marker, Mistral OCR, PyMuPDF4LLM, Docling, MarkItDown, and pdfToMarkdown.
Scoring dimensions
- Reading order.
- Table preservation.
- Heading hierarchy.
- OCR quality on scans.
- Markdown cleanliness.
- Setup time and operational complexity.
Publication rule
Only publish numeric scores after every tool has been run on the same input files, with raw outputs saved for review. Until then, comparison pages should describe tradeoffs and methodology, not claim a winner.