Improving confidence scores for reliable automatic processing of financial documents
Perception, Layout, and Validation: Calibrated Confidence for Reliable Straight-Through Processing of Financial Documents
Artificial Intelligence
Summary
Extracting important information from financial documents like invoices automatically is tricky because computers often aren’t sure if their answers are correct. The authors found that the natural confidence scores from language-vision AI models don’t reliably show when extractions are right or wrong. They designed a new method that looks at three parts—perception, document layout, and validation—to better judge confidence. This method significantly improves how well correct extractions can be separated from incorrect ones, allowing many more fields to be auto-approved safely without human checks.
What this means in practice
- •For financial software developers: Enhance automated data extraction pipelines for invoices with better confidence scores for safe auto-approval of fields.
- •For document automation teams: Increase efficiency by reliably auto-approving extracted fields from various financial forms without human review.
Authors
Yichao Jin, Yushuo Wang, Yuxuan Han, Kwan Ching Yee Sonia, Weiyang Song, Chiu Jin-Chun Kent, Wong Chong Hwee, Wong Tiong Kiat, Kenneth Zhu Ke, Jingyuan Zhao
Abstract
Straight-through processing (STP) on extracted key-value fields from financial documents without human review requires a calibrated probability together with a bounded guarantee on the residual error of the auto-approved tier. The emergence of modern Vision Language Models (VLMs) provides an out-of-the-box capability for extracting the key-values, but their verbalized confidence signals are unreliable and weakly track field correctness. This paper introduces a decomposed confidence layer along three interpretable channels, including perception, layout, and validation. Together with a final conformal risk control, the score can be used for reliable STP of financial documents. The method is validated on three public datasets covering real invoices, synthetic invoices, and ad-buy forms, using two different VLM families (Qwen3.6-27B and Gemini-3.1-Flash-Lite). Our decomposed score consistently improves the separation of correct from incorrect extractions, substantially raising the AUROC from 0.54-0.74 for VLM verbalized signals to 0.90-0.99 with contributions from all three designed channels. Crucially for industrial deployment, this enables usable STP. The native VLM confidence signals could clear only 0.1%-7.0% of fields under risk control at a target error of <10%. In contrast, the proposed method auto-approves 49-72% of fields while holding the empirical error of the accepted tier at or below the target.