OmniExtractBench is an open benchmark for structured document extraction that tests how accurately a system fills a JSON schema from a PDF. It combines 620 documents from four existing suites—ExtractBench, LongExtractBench, LongArray-Extract and Datalab’s internal synthetic set—covering forms, filings, decks, dense scalar schemas and large tables. The benchmark was created to solve four recurring problems with vendor‑specific leaderboards: bias toward the benchmark creator, opaque harnesses that hide implementation errors, unclear scoring that does not explain low scores, and narrow document variety that favors certain layouts.
The scorer works by flattening both prediction and gold JSON into address‑value pairs, normalizing scalar values (e.g., “03/31/2024” matches “2024‑03‑31”), then pairing table rows by content using the Hungarian algorithm instead of naïve positional comparison. Each paired value receives one of six auditable verdicts: matched, misread, unfound, fabricated, invented_item, or invented_field. Accuracy is the proportion of matched values over all verdicts, with precision and recall derived from matched versus predicted and gold values. A null rule treats empty strings, None and whitespace as omissions, preventing score inflation by padding schemas with empty optional fields.
To run the benchmark locally, install the scorer from PyPI (omni‑extract‑bench, Python 3.11+, SciPy only) and execute a command such as:
uv pip install omni-extract-bench
oeb score --pred pred.json --gt gold.json --schema schema.json --verdicts
Users must supply their own API keys and paid credits for vendor APIs; runs are resumable.
In Datalab’s evaluation, the accurate configuration achieved 93.85 % accuracy, while the balanced mode and Reducto deep_extract v2 tied near 93.5 %. Systems that lean toward misses (e.g., GPT 5.6‑sol, Gemini, Claude) show high precision but lower recall due to unfound values. Those that lean toward inventions (e.g., LlamaExtract, Extend) exhibit high recall but lower precision from fabricated or invented items. Low‑performing models trail on both metrics.
OmniExtractBench provides a transparent, comparable yardstick for measuring and improving PDF extraction models, enabling teams to pinpoint specific error types and track progress objectively.