Generalist specialist AI improves rare disease detection in medical images
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Detecting rare diseases from medical images is hard because each imaging type needs special knowledge, but sharing information across types can help. The paper shows that usual AI models with multiple experts don't balance this well. The authors propose a new model that uses both general knowledge across imaging types and specific knowledge for each type, working together. This approach better identifies rare diseases and uses fewer computing resources.
What this means in practice
- •For medical imaging ai developers: Improve detection accuracy of rare diseases across multiple imaging types by integrating generalist and specialist expert models with fusion methods.
- •For radiology it teams: Deploy more efficient AI systems that detect rare conditions with fewer active parameters, reducing computational costs during inference.
Authors
Johannes Kaiser, Florian Braunmiller, Daniel Rückert, Georgios Kaissis
Abstract
AI models for multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure Mixture-of-Experts (MoE) architectures currently fail to satisfy. Expert-based routing improves in-domain learning but may sacrifice cross-modal signals, which appear particularly important for rare (low-prevalence) pathologies in our experiments. To resolve this, we introduce Generalist-Specialist-MoE (GS-MoE), a two-branch (MoE) architecture that couples a cross-modal generalist model with distinct modality-specific specialists (experts) via domain-constrained feature fusion. On RadImageNet (1.35M images, 165 pathologies, three modalities), GS-MoE recovers detection of six low-prevalence pathologies on which every baseline scores F1 $=$ 0, with per-class gains up to +0.60 F1. It attains this while even slightly exceeding dense and specialist-only MoE aggregate baselines (MCC 0.770), while using ${\sim}53\%$ fewer active parameters at inference than the strongest investigated dense model.