Papers for

multimodal machine learning engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multimodal models can fail when some inputs go missing at test time

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

Abstract: Missing modality remains a longstanding challenge in multimodal learning. Existing methods typically address this issue through modality recovery or adaptive strategies. However, they overlook models' internal cross-modal dependencies formed during multimodal training, which later impair robustness. We systematically characterize a counterintuitive deployment-time failure mode: models trained on full modalities can underperform unimodal models when one modality is missing at inference time. This pattern appears across diverse architectures, such as fusion models, CLIP-style two-tower models, and vision-language models. We show that such degradation is closely associated with learned cross-modal dependencies in the principal parameter subspaces. Multimodal training induces structured rotations of these subspaces, particularly in cross-modal interaction layers. These rotations are associated with reduced task-aligned margins and larger task-aware representation harm under missing-modality inputs. We propose Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness. It rotates the principal input subspace toward a unimodal reference along a geodesic path. We prove that this correction minimizes the distance to the reference within a fixed subspace-distance budget. Experiments across architectures and datasets show that GU improves performance under missing-modality inference while preserving full-modality accuracy, outperforming strong missing-modality robustness baselines. These findings support a geometric view of deployment-time missing-modality degradation and suggest localized subspace editing as a practical route for robustness correction.

Sat 3 OctArtificial IntelligenceComputer Vision and Pattern Recognition
The gist
Multimodal learning combines data from different sources, like images and text, to improve performance. However, when some input types are missing during use, models trained on all inputs can do worse than those trained on only one type. The authors found this problem is linked to how the model’s internal parts depend on each other after training. They designed a new method called Geodesic Unlearning that adjusts the model to behave better when some inputs are missing, without losing accuracy when all inputs are available.
Open → 2610.04792v1

Multimodal model splits task signals from noise for better prediction

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Abstract: Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

Mon 28 SeptMachine Learning
The gist
When computers look at multiple types of input, like images and text together, they need to figure out which parts of these inputs really help solve a problem and which parts are just noise or unrelated. The authors created a new method that breaks down the information into parts that are shared across all inputs, parts unique to each input type, and irrelevant parts, all in a structured way within the computer’s internal reasoning. This helps the computer understand how each kind of input adds to solving the problem, leading to better predictions. They showed this works well on various tests with different types of data.
Open → 2609.35502v1