Pace moe improves accuracy in medical image and text matching
PACE-MoE: Adaptive Domain Sampling and Domain-Specific LoRA for Medical Visual Grounding
Computer Vision and Pattern Recognition
Summary
Matching medical images with the right clinical descriptions is difficult because there are many types of medical images. The authors developed PACE-MoE, a new system that learns which types of images to focus on during training and uses special adapters for each type. This approach helps the system better link medical text with the correct part of the image. Their tests show it works better than older methods across several kinds of medical images.
What this means in practice
- •For medical imaging developers: Improve accuracy of software that links clinical text to specific parts of diverse medical images using adaptive training strategies.
- •For radiology software teams: Enhance multimodal radiology tools by customizing model updates per imaging domain, improving detection and annotation consistency.
Authors
Zhenxuan Wang, Jisheng Dang, Ronghao Lin, Bimei Wang, Jingwen Zhao, Wenjun Ma, Bin Hu, Qi Tian, Tat-Seng Chua
Abstract
Medical visual grounding requires models to align clinical descriptions with spatial evidence across heterogeneous imaging domains. However, existing multi-domain post-training strategies typically rely on fixed domain sampling and shared parameter updates, overlooking the need to adapt both which domains receive training and where their updates are applied. To address this issue, we propose PACE-MoE, a policy-guided framework that jointly performs adaptive domain sampling and domain-specific adaptation for medical visual grounding. PACE dynamically assesses the training value of each imaging domain from current-policy feedback and progressively adjusts domain coverage through domain admission, replay, and final all-domain balancing. Meanwhile, domain-specific LoRA adapters route updates to the corresponding imaging domains while preserving the pretrained backbone and visual encoder. This coordinated design aligns training-data allocation with parameter adaptation without modifying the original reward formulation or Group Relative Policy Optimization (GRPO) objective. Experiments across seven medical imaging domains show that PACE-MoE consistently improves grounding performance over fixed-domain training, achieving a 3.64% relative improvement in mean box IoU over the Vanilla baseline. Under matched domain-specific adapter architectures, adaptive sampling alone further improves mean box IoU by 2.07%, demonstrating the benefit of policy-guided domain allocation. These results establish coordinated data-parameter routing as an effective strategy for multi-domain medical visual grounding.