Multiview semantically consistent encoding improves multi-label prediction with missing data
When Semantically Consistent Encoding Meets View-Label Heterogeneity Modeling: A Unified Framework for Incomplete Multi-View Multi-Label Learning
Computer Vision and Pattern Recognition
Summary
Some problems involve identifying multiple labels from data collected in different ways, but often some of these data views are missing or incomplete. The authors propose a new method called V2L that first creates a shared understanding from the partial data and then actively figures out which data sources are important for each label. This dual approach helps make more accurate predictions even when some information is missing. Their experiments show this method works better than previous ones on various benchmark tests.
What this means in practice
- •For computer vision engineers: Improve multi-label image tagging from incomplete multi-sensor or multi-angle data by adaptively combining available views for each label prediction.
- •For medical data analysts: Enhance disease prediction from partially missing multi-modal patient records by encoding consistent semantic features and weighting data sources per diagnosis.
Authors
Chengliang Liu, Bo Li, Bob Zhang, Yanghao Zhou, Jie Wen, Wenwu Wang
Abstract
Incomplete multi-view multi-label learning requires not only robust semantic aggregation from partially observed views, but also label-aware exploitation of view-specific evidence. Existing approaches usually emphasize either shared representation learning or decision-level fusion. The former improves robustness against missing views, yet tends to compress label-discriminative view-specific cues into a single latent representation. The latter preserves individual view predictions, but often relies on fixed or globally learned fusion weights, ignoring that different labels of different instances may require different views. To address these limitations, this paper presents V2L, a unified representation-decision framework for incomplete multi-view multi-label classification. On the representation side, V2L constructs semantically consistent variational posteriors from incomplete views through a perturbation-aware encoding mechanism, which provides a stable shared semantic basis. On the decision side, V2L introduces an active view-label relevance modeling strategy that estimates instance-wise and label-wise view contributions, allowing each label prediction to adaptively select useful view-specific evidence. From the perspective of model architecture, these two important strategies are integrated into a unified framework through a hybrid fusion architecture, simultaneously meeting the requirements of cross-view semantic consistency and representational complementarity. Extensive experiments under both incomplete and complete settings show that V2L achieves leading performance on five benchmarks. Code is available at: https://github.com/justsmart/V2L.