Vision language models trade off accuracy cost and governance in document extraction
Beyond Accuracy: Robustness, Cost, and Governance Trade-offs for Vision-Language Models in Templated Document Extraction
Artificial Intelligence
Summary
Extracting important information from business documents often uses computer programs that understand both images and text. This study compared eleven different systems on a set of 750 sample documents and found that some open-source models, when trained with enough examples, performed very well — even better than several commercial systems. The researchers also noticed that the performance can vary a lot depending on the specific task, like extracting dates. They created a practical guide to help people choose the best approach based on quality, speed, rules, and cost.
What this means in practice
- •For document processing teams: Select document extraction methods that balance accuracy, cost, and governance based on specific workload and quality needs.
- •For business automation engineers: Use fine-tuned open-source vision-language models to improve accuracy on extracting fields from complex business documents compared to default commercial tools.
Authors
Kushal Patel, Pushkal Shrivastava, Mackenzie Lees, Qirui Lu, Bhargobjyoti Saikia, Liying Li, Junlin Jiang
Abstract
Vision-language models (VLMs) are increasingly used to extract structured fields from business documents, yet most evaluations report accuracy on clean benchmarks and offer little guidance to practitioners choosing an approach for a given task complexity. We address this gap with a measurement-grounded study and an open-source release. Across eleven systems (three commercial, two reasoning, five open-source VLMs in pretrained and fine-tuned form, and a non-LLM OCR->regex floor) scored on a 750-document held-out pool of synthetic checks, fine-tuning on 3K samples lifts the best open-source VLMs above F1 0.98-above every zero-shot commercial system on this task-while GPT-5 leads the commercial pool on F1 and Claude Sonnet 4.5 collapses on Date. To turn these measurements into actionable choices, we introduce a practitioner-oriented selection framework that maps a task profile (quality, latency, governance, volume) to a recommended approach via filtering and total-cost minimization, illustrated on a hypothetical mid-volume document-extraction scenario.