Multimodal Semantic-Aware Contrastive Learning For False Negative Mitigation in 3D Medical Imaging
Machine Learning
Summary
The authors studied a way to improve how computer programs learn from different types of medical data, like brain scans and doctor reports, by making the learning aware of the meaning behind the reports. Usually, such learning tries to match pairs of data and push apart unmatched pairs, but sometimes unmatched samples are actually similar, which causes confusion. They created a method called MseaCL that uses the similarity between reports to avoid wrongly treating related data as negative examples. This helped their program do better at diagnosing pediatric brain tumors from brain scans, showing it can make medical data understanding more accurate.
Authors
Sara Ketabi, Matthias W. Wagner, Cynthia Hawkins, Uri Tabori, Birgit Betina Ertl-Wagner, Farzad Khalvati
Abstract
Multimodal Contrastive Learning (CL) has shown significant performance in aligning representations across various data modalities and improving downstream tasks, especially in healthcare. It works by minimizing the distance between matched (positive) data modalities, while maximizing the distance between mismatched (negative) samples. Traditional CL frameworks typically assume instance-based correspondence within data batches, treating all non-paired samples as negatives. However, this assumption often fails in medical settings, where samples may share high-level semantic attributes, leading to false negatives that degrade representation quality. In this paper, we propose Multimodal Semantic-Aware Contrastive Learning (MseaCL), a CL framework trained on a pediatric cohort of 3D brain magnetic resonance imaging (MRI) scans and radiology reports. The goal of this framework is to mitigate the impact of semantically similar false negative samples by incorporating semantic similarity between radiology reports, as a guiding signal during the learning process. Our results indicate that applying this framework as a pretraining stage can achieve notable improvements in downstream tasks, e.g., at least a 22.6\% increase in the area under the receiver operating characteristic curve (AUC) of pediatric brain tumor molecular classification, demonstrating its potential for more robust and semantically aligned multimodal representations in clinical applications.