System reads documents and solves math expressions with mixed accuracy
Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem
Computation and LanguageInformation Retrieval
Summary
Some documents have questions that require finding text and doing math to answer them. The authors made a system that looks at PDF pages, picks text passages, asks a language model to create arithmetic expressions, and then calculates the answers. It worked very well on a small public set but less well on a larger test set, partly due to issues like text being mixed up during scanning. Their analysis shows some reasons for errors but doesn’t explain all of them.
What this means in practice
- •For document processing teams: Automatically extract numerical answers from scanned PDFs by combining text selection and arithmetic evaluation with a language model.
- •For quality assurance teams: Use saved intermediate results and evidence identifiers to inspect and debug failures in document-based question answering systems.
Authors
Divya Godara, Sachin Gupta
Abstract
EVICALC, our system for the DocSem shared task, achieved 8.61% joint accuracy on 1,730 tasks in the official final test evaluation. It reads a PDF, selects a passage, asks a language model to write an arithmetic expression, and evaluates that expression in local code. Saved intermediate results support inspection of failures. A separate public-validation run achieved 92.17% answer accuracy and 1.00 evidence F1. The configurations and metrics differ, so these scores are not a controlled comparison. Our manual, post-hoc analysis is descriptive: in one inspected case, optical character recognition (OCR) and block grouping merged the relevant passage into another block, and the system answered from unrelated text. An exploratory study of reading page images on 100 documents returned evidence identifiers for only 22 documents. These descriptive findings motivate further evaluation; they do not establish the causes of the overall score.