Multimodal learning improves trust by detecting noisy labels better
Learning What to Trust in Multimodal Learning under Noisy Supervision
Computer Vision and Pattern RecognitionMachine Learning
Summary
Multimodal learning combines different types of information to make better predictions, but it usually needs very correct labels which are hard to get. The paper studies how to better find wrong labels by using both combined and separate data views, making it easier to tell which labels to trust. The authors propose a method called REFINE that picks out cleaner data for training, leading to more accurate results. Their tests show that REFINE works better than older methods on various tasks.
What this means in practice
- •For machine learning engineers: Improve training of classification models by detecting and removing mislabeled examples in multimodal datasets.
- •For data annotation teams: Identify unreliable labels in multimodal datasets to reduce labeling errors and improve data quality.
Authors
Jiashuo Zou, Xiaobo Xia
Abstract
Multimodal classification processes and relates information from multiple modalities to achieve more accurate predictions. However, existing methods typically rely on high-quality ground-truth labels, which are difficult to obtain in real-world scenarios. While sample-selection methods for learning with noisy labels aim to identify correctly labeled examples from noisy data, traditional methods primarily focus on unimodal settings and fail to exploit multimodal information fully. This motivates us to build a more reliable noise detector in multimodal learning. To this end, we theoretically analyze the relationship between representation structure and noise detection capability. Based on this analysis, we propose REFINE, which is a multimodal label-noise detection framework that jointly uses fused and unimodal representations for label-noise detection. Specifically, REFINE constructs discriminative eigenvectors through discriminative analysis of the target and background classes and selects trusted representation spaces with better noise detection capability for each class. Within each trusted space, REFINE measures the alignment between each instance representation and the discriminative eigenvectors. It then combines the subsets selected from these spaces. The combined set provides cleaner supervision for updating the multimodal classifier, thereby reducing the influence of mislabeled examples during training and improving model generalization. Extensive experiments across diverse tasks demonstrate REFINE's superiority compared to baseline methods. The source code will be publicly available.