Multimodal models can fail when some inputs go missing at test time
Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Multimodal learning combines data from different sources, like images and text, to improve performance. However, when some input types are missing during use, models trained on all inputs can do worse than those trained on only one type. The authors found this problem is linked to how the model’s internal parts depend on each other after training. They designed a new method called Geodesic Unlearning that adjusts the model to behave better when some inputs are missing, without losing accuracy when all inputs are available.
What this means in practice
- •For multimodal machine learning engineers: Improve model accuracy when some input data types are unavailable during deployment by applying geometric subspace corrections.
- •For software developers of ai-powered apps: Maintain reliable performance of vision-language features even when some sensors or input channels fail or are turned off.
Authors
Songyuan Sui, Zhen Tan, Mohan Zhang, Rana Muhammad Shahroz Khan, Xia Hu, Tianlong Chen
Abstract
Missing modality remains a longstanding challenge in multimodal learning. Existing methods typically address this issue through modality recovery or adaptive strategies. However, they overlook models' internal cross-modal dependencies formed during multimodal training, which later impair robustness. We systematically characterize a counterintuitive deployment-time failure mode: models trained on full modalities can underperform unimodal models when one modality is missing at inference time. This pattern appears across diverse architectures, such as fusion models, CLIP-style two-tower models, and vision-language models. We show that such degradation is closely associated with learned cross-modal dependencies in the principal parameter subspaces. Multimodal training induces structured rotations of these subspaces, particularly in cross-modal interaction layers. These rotations are associated with reduced task-aligned margins and larger task-aware representation harm under missing-modality inputs. We propose Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness. It rotates the principal input subspace toward a unimodal reference along a geodesic path. We prove that this correction minimizes the distance to the reference within a fixed subspace-distance budget. Experiments across architectures and datasets show that GU improves performance under missing-modality inference while preserving full-modality accuracy, outperforming strong missing-modality robustness baselines. These findings support a geometric view of deployment-time missing-modality degradation and suggest localized subspace editing as a practical route for robustness correction.