Lluna improves performance when some input data is missing
Lluna: Collaborative Unimodal Models for Missing-Modality Inference
Multimedia
Summary
Many computer programs use different types of data together to make predictions, but sometimes some types of data are missing, which makes these programs less accurate. The authors study how models that work with single types of data can still collaborate effectively even when some data is missing. They propose Lluna, a new method that combines private predictions with collaboration that adjusts for missing data. Lluna performs better than previous methods on tasks like understanding emotions, scenes, and medical predictions when some input is unavailable.
What this means in practice
- •For healthcare data teams: Improve medical predictions when some patient data types are missing by using Lluna's collaborative approach to combine available signals for more accurate results.
- •For computer vision teams: Enhance scene understanding systems to maintain performance even when some sensor inputs are unavailable by using Lluna's missing-modality inference method.
Authors
Haoyue Bai, Yinyi Luo, Heqing Huang, Xiaoyuan Yi, James Evans, Jindong Wang
Abstract
Pretrained unimodal models offer a flexible route to multimodal prediction through collaboration, but this collaboration can become less accurate than a surviving unimodal model when modalities are missing. We study this setting using fixed surviving-model anchors and find that collaborative gains can weaken or reverse as inputs disappear, even when the anchor's own input is unchanged. Our diagnostics further reveal reduced recovery of modality-specific information after aggregation and state-dependent collaborative representations. We propose Lluna, which combines a direct private prediction path from adapted unimodal features with a common graph-based collaboration branch that provides a residual correction. Progressive Collaboration Consistency organizes richer-to-poorer guidance along single-modality removals, aligning normalized common representations while task supervision adapts predictions to the available evidence. Across eight datasets spanning affective computing, scene understanding, and medical prediction, Lluna achieves the best evaluated average performance over non-full input states, improving Mean-Missing by 4.26\% on average relative to the strongest evaluated baseline. The fixed-anchor diagnostic also shows positive gains in mean collaboration under minimal inputs on all eight tasks. Code is available at https://github.com/AIFrontierLab/Lluna.