Reliability-aware fusion improves learning with missing multimodal data
Reliability-Aware Probabilistic Fusion for Multimodal Learning with Incomplete Evidence
Machine Learning
Summary
Sometimes data from multiple sources can be missing or low quality, which makes it hard for computer programs to make good decisions. The authors created a method that weighs how reliable each piece of information is before combining it. This approach works especially well when some data is missing, like in medical tasks with incomplete patient records. They also tested what happens when data is corrupted and found their method can adjust but struggles if the data is very bad.
What this means in practice
- •For clinical data teams: Improve patient outcome predictions when some clinical data modalities are missing or unreliable.
- •For autonomous vehicle engineers: Combine sensor inputs that may be partially missing or degraded while adaptively weighting their reliability.
Authors
Patitapaban Palo, Abhisek Tiwari, Mattia Carletti, Anshul Thakur, Xiao Gu, David A. Clifton
Abstract
Multimodal inputs may be missing or degraded, raising distinct questions of evidence availability and trust. We introduce a reliability-aware Gaussian Product-of-Experts (PoE) in which an input-dependent reliability weight tempers each modality's latent precision, while missing experts are omitted. We characterize how reliability and availability affect posterior information and expert influence. Across two clinical multimodal tasks, the full framework remains competitive with complete inputs and achieves its clearest mean-AUROC advantage under modality absence, particularly on HAIM-MIMIC-MM. Ablations show a substantial gain from missing-aware training and an additional HAIM improvement from reliability weighting under matched PoE, missing-aware, and consistency training. Unseen test-time corruptions are used diagnostically and reveal modality-dependent reliability-weight and latent-uncertainty responses, including failure under severe degradation. Thus, reliability weighting provides an explicit evidence-control mechanism, while learning dependable quality signals remains an open challenge.