WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created a new test called WildHandBench to better check how well computer models can read handwritten documents, including text, tables, and formulas in different languages and real-world settings. They found that the best models are much worse at this task than reading printed documents, scoring about 72% accuracy, while humans do slightly better at 77%. They also introduced a new way to measure errors that shows models often guess based on language expectations rather than actually reading the handwriting, unlike humans whose mistakes are less biased. This suggests current models rely too much on guessing the words instead of understanding the handwriting itself.
handwritten document parsingbenchmarkshandwriting recognitionlanguage priorserror analysistables recognitionformulas parsingaccuracy metricsmachine learning modelshuman baseline
Authors
Jun Zhang, Qiao Zhao, Cheng Cui, Jianying Qu, Zhongkai Sun, Jianwen Yang, Changda Zhou, ZhuoXin Liu, Shubin Han
Abstract
While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors -- 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture.