HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors study how to improve vision-language models for chest X-rays by using multiple labeled datasets instead of relying mostly on one big source of image-text pairs. They propose a new method called HarMoE that helps the model learn shared medical concepts while handling differences between datasets separately. This approach lets the model better combine clean disease labels from many sources without confusion, improving accuracy on various tasks even without extra training on target data. Their experiments show that using diverse, well-organized datasets leads to more reliable and generalizable chest X-ray understanding.
vision-language modelschest X-raymulti-label classificationdataset harmonizationmixture-of-expertszero-shot classificationout-of-distribution transfermedical image analysisdisease annotationcross-dataset learning
Authors
Haozhe Luo, Ziyu Zhou, Shelley Zixin Shu, Mauricio Reyes
Abstract
Recent vision-language models for chest X-ray understanding are largely built on image-report alignment and therefore rely heavily on MIMIC-CXR as the dominant pretraining source. While effective at scale, this paradigm underexplores an important alternative source of supervision: a range of existing multi-label classification datasets, which provide cleaner and more explicit disease signals than free-text reports, and can offer broader pathology coverage when combined across sources. However, learning from such heterogeneous datasets is nontrivial, as differences in label ontologies, annotation protocols, acquisition pipelines, and report styles can cause models to entangle clinical semantics with dataset identity, leading to poor transfer despite increased scale. In this work, we revisit radiology VLM construction from the perspective of harmonized multi-source learning. We propose HarMoE, a dataset-aware mixture-of-experts framework that learns shared cross-dataset medical semantics while confining source-specific variation to lightweight residual experts in deeper decoder layers. To further exploit clean supervision from labeled datasets, we train in a unified disease vocabulary with masked multi-dataset supervision, enabling the model to leverage complementary annotations without introducing false negatives. Experiments on large-scale chest X-ray benchmarks show that HarMoE consistently improves zero-shot classification, out-of-distribution transfer, and grounding over strong baselines. Our results suggest that building robust radiology VLMs requires moving beyond single-source image-report alignment toward structured knowledge construction from heterogeneous datasets with cleaner supervision and broader coverage. Code and the 873k harmonized dataset will be released at https://github.com/Roypic/harmoe.