Multimodal Approaches for Visually-Rich Document Type Classification: A Comparative Analysis
2026-06-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and LanguageInformation Retrieval
AI summaryⓘ
The authors studied how to best classify types of documents that have important information spread across text, images, and layout features. They compared four popular models using a consistent testing setup to see how text, images, and layout affected performance. They found that specialized multimodal transformers worked better than large language models, especially when documents had complex layouts. Images were the most helpful feature, while OCR-extracted text helped too but was less important. Their work helps clarify which features and model designs are most effective for understanding visually rich documents.
multimodal modelingtransformerslarge language modelsOCRdocument classificationvisual layout analysisRVL-CDIP benchmarkLayoutLMv3Donut modelfeature fusion
Authors
Catyana Heyne, Jürgen Frikel, Filippo Riccio
Abstract
Document type classification in visually rich documents remains challenging, as relevant information is distributed across textual, visual, and layout modalities. To capture this complexity, current approaches rely on diverse multimodal modeling strategies, resulting in heterogeneous architectures that complicate systematic comparison. This variability is also reflected in existing comparative studies, which often rely on heterogeneous evaluation setups, further complicating systematic comparison and making it difficult to assess progress. To address these limitations, this work provides a structured analysis of multimodal design strategies across transformer- and LLM-based architectures, combined with a controlled empirical comparison within a unified experimental framework. Specifically, four representative models (LayoutLMv3, Donut, Qwen3-VL-32B-Instruct, and Qwen3-32B) are evaluated on the RVL-CDIP benchmark to systematically analyze the contributions of text, image, and layout information for document type classification, with a particular focus on contrasting OCR-dependent and OCR-free approaches. The results show that specialized multimodal Transformers outperform LLM-based approaches on visually rich and layout-intensive documents. Image information contributes most strongly to reliable classification, while OCR-derived text provides useful but secondary support. These findings highlight that multimodal processing remains essential for documents with pronounced layout structure. Overall, the study provides a systematic basis for comparing multimodal architectures and offers practical guidance for selecting effective feature combinations and model designs for document type classification.