Entropy improves answer tracing and accuracy in multimodal question answering
EDMA: Entropy-Driven Multimodal Answering
Artificial Intelligence
Summary
Multimodal question answering means getting answers by combining different types of information, like text and pictures. The authors say that measuring how much each type of information helps find the right answer can make the process clearer and better. They created a method called EDMA that uses a concept called logical entropy to track and reduce uncertainty step by step when picking the final answer. Their tests show this approach helps computers answer questions more accurately and explains why certain information mattered in the decision.
What this means in practice
- •For software developers: Enhance AI assistants with clearer, more accurate answers by tracking how different input types contribute to decisions.
- •For data integration teams: Improve system design by quantifying and combining data from multiple sources to reduce errors in decision-making.
Authors
Emanuele Mezzi, Gertjan Burghouts, Fabio Massacci, Mengyuan Zhang
Abstract
Multimodal question answering (MMQA) is based on integrating heterogeneous data sources, selectively leveraging relevant modalities, and ignoring distractors. Existing approaches based on Multimodal Large Language Models (MLLMs) improve performance by splitting the task into intermediate steps, but do not quantify how each modality contributes to reaching the final answer. We argue that quantifying the effect that each modality has on the answering process improves process traceability and final performance. Thus, we propose Entropy-Driven Multimodal Answering (EDMA), which uses logical entropy to formalise multimodal QA as an iterative process of entropy reduction and to quantify the contribution of each modality in distinguishing correct from incorrect candidate answers. This enables the construction of quantifiable answering trajectories, where each step is associated with a measurable entropy reduction. EDMA introduces (i) modality-conditioned partitioning to estimate unimodal entropy reduction, (ii) entropy-driven multimodal fusion to capture complementary information across modalities, and (iii) entropy-based selection of the answering trajectory that minimises the most logical entropy. Evaluated across standard MMQA benchmarks, EDMA outperforms competitive prompting baselines, with average gains of 12.8 F1 and 4.6 LLM-J points over Planning Prompting and 18.6 F1 and 8.3 LLM-J points over Chain-of-Thought. Moreover, EDMA outperforms self-consistency baselines and is competitive with the multi-agent system baseline. Moreover, we show that higher entropy correlates with more false positives and, through controlled trajectory selection, demonstrate that logical entropy serves as an effective, monotonic selector for correct reasoning paths, validating entropy level as a reliable proxy for performance.