Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesis
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created SAYRE, a system that makes fake but realistic documents to help train models that can pull important info from complex documents without needing lots of manual templates. SAYRE learns patterns from a few examples and makes new training data, including tricky cases where models usually mess up. Their tests show SAYRE helps improve smaller models running locally and works better for different types of documents, especially with tricky details like tables and contract terms. This approach focuses on better training data to make extracting key info from documents easier and more accurate.
Key Information Extraction (KIE)Large Multimodal Models (LMMs)Document synthesisScene-aware generationTraining data augmentationSchema annotationError-driven learningOn-device modelsLayout conventionsDense tables
Authors
Zhipeng Xu, Zulong Chen, Qing Liu, Junhao Ji, Jinxin Hu, Yipeng Yu, Jianqiang Wan, Jun Tang, Zhao Li
Abstract
Key Information Extraction (KIE) converts visually rich documents into structured data, but practical deployment remains challenging: strong performance often relies on costly on-server Large Multimodal Models (LMMs), while compact locally deployable models lack sufficient KIE supervision. We present SAYRE, a scene-aware document synthesis framework for generating scalable KIE training data without hand-crafted template design. Given a few exemplar documents, SAYRE captures category-specific content patterns and layout conventions to synthesize document-schema-annotation triples. It further introduces error-driven generation, which expands real-world failure cases into hard training examples while preserving their structural patterns. Experiments on constrained- and open-category KIE show that SAYRE consistently improves Qwen3-VL backbones and achieves the strongest overall performance among on-device LMMs. Data scaling experiments show an overall upward trend as more synthesized data is introduced, especially for smaller models and open-category extraction. Error analysis further shows that synthesized training reduces field-level errors by improving schema-aware extraction over dense tables, business identifiers, and contract clauses. These results establish scene-aware synthesis as an effective data-centric approach for improving practical multimodal KIE.