OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created a new test called OCR-MetaReasoning to check how well AI models can understand and reason about text in images. Unlike previous tests, this one separates whether the model gets the final answer right from whether it follows the right reasoning steps, focusing on three kinds of reasoning: deduction, induction, and abduction. They tested various models and found that many struggle to apply visible rules or understand the layout of the text, and sometimes models give the right answer but with flawed reasoning. Their benchmark provides tools to measure both answer accuracy and reasoning process compliance.
Multimodal Large Language ModelsOCRDeductionInductionAbductionReasoning ProcessLayout AnalysisBenchmark DatasetMeta-ReasoningAnswer Scoring
Authors
Gengxu Li, Yuan Wu, Yi Chang
Abstract
Text-rich image understanding requires multimodal large language models (MLLMs) to organize OCR (Optical Character Recognition)-grounded evidence across words, layout, fields, charts, and visual correspondences. Existing evaluations often conflate extraction with reasoning and rarely test whether models follow the required reasoning direction: applying visible rules, abstracting hidden regularities, or recovering missing premises. We introduce OCR-MetaReasoning, a controlled single-image benchmark that treats deduction, induction, and abduction as distinct directions and separates final-answer correctness from reasoning-process compliance. The benchmark contains 1,500 verified samples in a balanced \(3\times5\) taxonomy crossing three reasoning types with five OCR-object categories, along with reference reasoning steps, automatic answer scoring, the Meta-Reasoning Macro Score (MRMS), and the Reasoning Process Compliance Score (RPCS). Experiments with representative closed-source and open-source MLLMs show that OCR-grounded meta-reasoning remains far from saturated: models struggle with visible-rule application and layout-sensitive inference, while process-compliant rationales can accompany incorrect final answers under exact-match evaluation. The code is available at https://github.com/gengxuli/OCR-MetaReasoning.