Multimodal adaptation improves AI reliability with missing or noisy data
Calibrating Multimodal Test-Time Adaptation under Reliability and Availability Modality Shifts
Computer Vision and Pattern Recognition
Summary
Multimodal AI systems use different types of information like images and text to make predictions. The researchers studied how to help these systems keep working well when some types of information become unreliable or are missing during use. They created a method called AdaPGC that adjusts AI predictions on the fly by modeling and correcting uncertainties across these multiple data types. This helps the AI stay accurate even if some inputs get scrambled or disappear.
What this means in practice
- •For autonomous vehicle teams: Enhance sensor fusion algorithms to maintain safe perception when some sensors are noisy or fail.
- •For security camera system engineers: Improve video and audio analysis when either signal is degraded or temporarily unavailable.
Authors
Jiangmeng Li, Jinglin Xu, Yi Li, Yumeng Liu, Jiahuan Zhou, Fanjiang Xu, Hui Xiong, Gang Hua
Abstract
Multimodal test-time adaptation (TTA) enables pretrained multimodal models to accommodate distribution shifts during inference using only unlabeled test data, without recourse to the original training data. Compared with unimodal adaptation, multimodal adaptation is further complicated by the coexistence and interaction of multiple representation perspectives. This complexity is particularly pronounced under modality reliability shifts, which preserve modality availability but degrade information quality, and modality availability shifts, which render the corresponding modality entirely inaccessible. Category-conditional Gaussian modeling offers a natural starting point for handling such distributional variations, having proven effective in unimodal test-time adaptation for characterizing representation distributions, yet its extension to multimodal settings requires a more elaborate formulation. To this end, we propose Adaptive Probabilistic Gaussian Calibration (AdaPGC), which extends category-conditional Gaussian modeling across modality-specific and multimodal fusion representations for probabilistic calibration under multimodal distribution shifts. AdaPGC continuously updates category-conditional Gaussian models over modality-specific and fusion representations, using the resulting probabilistic predictions to calibrate source model predictions and guide reliability based rectification, while exploiting their joint distributions to recover unavailable fusion representations. Extensive experiments under diverse modality corruption and missingness conditions demonstrate that AdaPGC effectively improves robustness under both modality reliability and availability shifts. Our source code will be released at https://github.com/XuJinglinn/AdaPGC-plus.