Papers for

software developers of ai-powered apps

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multimodal models can fail when some inputs go missing at test time

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

Abstract: Missing modality remains a longstanding challenge in multimodal learning. Existing methods typically address this issue through modality recovery or adaptive strategies. However, they overlook models' internal cross-modal dependencies formed during multimodal training, which later impair robustness. We systematically characterize a counterintuitive deployment-time failure mode: models trained on full modalities can underperform unimodal models when one modality is missing at inference time. This pattern appears across diverse architectures, such as fusion models, CLIP-style two-tower models, and vision-language models. We show that such degradation is closely associated with learned cross-modal dependencies in the principal parameter subspaces. Multimodal training induces structured rotations of these subspaces, particularly in cross-modal interaction layers. These rotations are associated with reduced task-aligned margins and larger task-aware representation harm under missing-modality inputs. We propose Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness. It rotates the principal input subspace toward a unimodal reference along a geodesic path. We prove that this correction minimizes the distance to the reference within a fixed subspace-distance budget. Experiments across architectures and datasets show that GU improves performance under missing-modality inference while preserving full-modality accuracy, outperforming strong missing-modality robustness baselines. These findings support a geometric view of deployment-time missing-modality degradation and suggest localized subspace editing as a practical route for robustness correction.

Sat 3 OctArtificial IntelligenceComputer Vision and Pattern Recognition
The gist
Multimodal learning combines data from different sources, like images and text, to improve performance. However, when some input types are missing during use, models trained on all inputs can do worse than those trained on only one type. The authors found this problem is linked to how the model’s internal parts depend on each other after training. They designed a new method called Geodesic Unlearning that adjusts the model to behave better when some inputs are missing, without losing accuracy when all inputs are available.
Open → 2610.04792v1