Reliability-aware fusion improves predicting image metadata under uncertainty
RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion for Multimodal Prediction under Modality Uncertainty
Machine LearningComputer Vision and Pattern Recognition
Summary
Predicting information from images combined with other data sources can be tricky when some data is less reliable. The authors created a method called RiVaT-Fuse that blends image and metadata by estimating a shared hidden state, considering how trustworthy each source is for every example. This clever approach uses mathematical tools to better handle uncertainty and interactions between data, leading to more stable and accurate predictions than existing methods. Their approach works well on benchmarks, showing improved prediction and consistency when data is noisy.
What this means in practice
- •For computer vision engineers: Create image analysis systems that combine metadata robustly despite uncertain or noisy data sources.
- •For multimedia system designers: Develop multimodal prediction models that maintain stable outputs when input modalities vary in reliability.
Authors
Yingfan Xu, Tieming Liu, Ye Liang, Taiping Liu
Abstract
Image-metadata prediction requires fusing heterogeneous evidence whose reliability can vary across samples and latent factors. Existing representation-level fusion methods typically choose an aggregation architecture, such as concatenation, gating, conditional modulation, or attention, without explicitly defining what the fused representation should mean under modality uncertainty. We propose RiVaT-Fuse, a reliability-calibrated variational tensor fusion framework that defines fusion as sample-wise latent-state estimation. Rather than producing a fused vector by direct aggregation, RiVaT-Fuse estimates a consensus latent state through a variational objective that balances image evidence, metadata evidence, structured cross-modal interaction, and stability. The resulting framework replaces scalar modality confidence with matrix-valued trust geometry, decomposes interaction into additive, multiplicative, and relational components, and couples the latent state with conditional robustness and structured multi-task prediction. We provide well-posedness and stability interpretations of the latent solve and instantiate the framework with efficient low-rank-plus-diagonal trust operators. On an image-level image-metadata prediction benchmark, RiVaT-Fuse achieves the strongest overall predictive rank among direct representation-level baselines while improving probability and label stability under perturbation.