Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction
2026-08-03 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors created ConfBench, a new test set designed to check how well vision-language models can estimate their confidence in extracting information from documents, especially when documents are damaged or unclear. They tested several models and found that combining OCR and images helps confidence estimates, and better models give more reliable confidence scores. They also discovered ways to adjust these confidence scores to make automatic vs. human review decisions more effective. Additionally, they introduced a new metric called ECARB to measure how improved confidence leads to saved review work. ConfBench is made available publicly to help researchers improve trustworthy document processing.
Intelligent Document Processing (IDP)Vision-Language Models (VLMs)CalibrationConfidence EstimationKey Information Extraction (KIE)Optical Character Recognition (OCR)Post-hoc CalibrationDegradation PipelinesEvaluation MetricsOperational Savings
Authors
Priyashree Roy, Sujitha Martin, Mohammad Rostami, Spencer Romo, Renhao Xue, Bob Strahan, Diego A. Socolinsky, Boyi Xie, Md Mofijul Islam
Abstract
Intelligent document processing (IDP) with vision-language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between automation and human review. Existing document benchmarks are dominated by clean, high-quality samples, leaving low accuracy regions too sparse for calibration assessment. We introduce ConfBench, the first calibration-specific benchmark for key information extraction (KIE), built by applying 20 controlled degradation pipelines to a diverse document set, yielding 1,346 variants and 70K+ entity-level evaluations spanning the full accuracy spectrum. We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods across three input modalities, and find: (i) OCR+Image modality results in more accurate confidence estimates; (ii) model capability is the dominant factor: within the Claude family confidence quality scales monotonically with capability, while across families parameter count is a poor predictor; (iii) calibration quality varies widely across models, from near-perfect to severely overconfident, and per-model post-hoc correction rescales these absolute confidence values for threshold-based routing without altering ranking-based operational metrics; and (iv) log-probability with first-token aggregation consistently outperforms mean-token and margin aggregations. We also introduce ECARB, a review-budget metric translating discriminative gains into operational savings. We release ConfBench publicly to enable systematic study of confidence estimators and calibration methods for trustworthy IDP application deployment.