Multimodal question answering moves toward unified language models
Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation
Computation and LanguageComputer Vision and Pattern Recognition
Summary
Answering questions from mixed sources like text, tables, and pictures is getting better. The authors compare three different approaches, showing how systems have evolved from handling each type of data separately to using a single language-based model for everything. They find that this unified approach improves accuracy but also point out some challenges like losing details when changing data formats. Their work helps us understand how these systems work and where future improvements may be needed.
What this means in practice
- •For customer support teams: Improve automated response systems by integrating text, table, and image data for more accurate answers to customer questions.
- •For data analytics teams: Combine diverse data formats in dashboards to enable unified querying and reasoning across texts, tables, and images for deeper insights.
A survey. It maps existing work.
Authors
Abdullah Al Shafi
Abstract
The rapid growth of multimodal data has intensified the need for question answering (QA) systems capable of reasoning across heterogeneous sources such as text, tables, and images. In this paper, we present a comprehensive methodological comparison of three influential frameworks, namely Multimodal Adaptive Extraction (MAE), Solar, and UniMMQA, tracing the evolution of multimodal question answering from modality-adaptive pipelines to fully unified architectures. We examine how each approach models cross-modal interactions, transforms heterogeneous inputs, and performs reasoning, highlighting key design differences in modality representation, reasoning, and answer generation. Our analysis demonstrates a clear shift from explicit modality-specific processing toward unified text-centric formulations enabled by pre-trained language models (PLMs). Empirical comparisons across benchmark datasets show that this transition leads to substantial improvements in both Exact Match (EM) and F1-Scores, with UniMMQA achieving the most consistent and scalable performance. Despite these advances, we identify persistent challenges, including information loss during modality transformation, error propagation in multi-stage pipelines, and limitations in capturing fine-grained cross-modal dependencies. Overall, this study provides a deeper understanding of current design trends and offers insights into the future direction of unified multimodal reasoning systems.