Radiology report generation improves by aligning image and text features
Homogeneous Semantic Alignment and Hierarchical Expert Routing for Radiology Report Generation
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
Doctors often need detailed reports from medical images to help diagnose patients, but writing these reports takes time. This paper describes how the authors made a computer program that links images and related words more closely, so it understands medical pictures better. Their program uses a two-step process to focus on important visual clues and pick the best information for each report. Tests show this method creates more accurate and detailed radiology reports than previous ones.
What this means in practice
- •For medical imaging software developers: Build more accurate automatic radiology report systems by using aligned visual-text representations and expert routing for diverse clinical information.
- •For healthcare analytics teams: Enhance diagnostic data extraction from medical images by integrating purified visual features with multi-source clinical evidence for better decision support.
Authors
Erjian Zhang, Jiayuan Ma, Liejun Wang, Yikemaiti Sataer, Xiaoming Tao, Zhiqing Guo
Abstract
Radiology report generation (RRG) aims to convert medical images into diagnostic texts to assist in clinical decision-making and alleviate the workload of physicians. Although existing methods have made extensive progress in cross-modal interaction and the incorporation of external priors, the distribution shift of underlying representations and the undifferentiated rigid coupling of heterogeneous information cause weak visual abnormality cues to be easily diluted by massive text priors and generation inertia during decoding. To overcome this bottleneck, inspired by cognitive science, we propose a novel two-stage Homogeneous Semantic Alignment and Hierarchical Expert Routing (HSA-HER) framework. First, the model introduces an explicit homogeneous distribution constraint in the underlying latent space to effectively eliminate the cross-modal distribution shift between visual and textual features, thereby extracting purified visual features as semantic anchors that accurately align with diseases. Second, for heterogeneous clinical evidence composed of visual features, local entities, and global retrievals, we design a hierarchical expert routing mechanism guided by these disease semantic anchors. This mechanism abandons the undifferentiated rigid coupling paradigm. Specifically, it dynamically activates expert networks to perform targeted mining and semantic reconstruction on multi-source evidence, and adaptively allocates fusion weights. Extensive experiments on three mainstream benchmark datasets demonstrate that HSA-HER achieves state-of-the-art performance, accurately depicting complex imaging details and key diagnostic information.