Medical vision language model improves multi-task clinical image analysis
Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning
Computer Vision and Pattern Recognition
Summary
Medical images can be analyzed for many different tasks like diagnosing diseases or creating reports, but training one model to do all tasks well is hard. The authors propose a two-step method that uses shared and task-specific parts to learn these tasks together better. They first train everything together and then fine-tune each task separately. Their method showed good results on various medical imaging tests.
What this means in practice
- •For clinical ai developers: Build systems that analyze medical images across several tasks with improved accuracy and balance among different outputs.
- •For medical imaging software teams: Enhance diagnostic and reporting functions by integrating multi-task vision-language learning for varied medical imaging data.
Authors
Zhanghao Chen, Yuanyuan Li, Zhenyu Lu, Shuo Gao, Guangquan Zhou, Yikun Zhang
Abstract
Medical vision-language models (VLMs) allow a single model to perform clinical image analysis tasks ranging from diagnosis classification to report generation. However, joint adaptation is challenged by heterogeneous output formats, conflicting task gradients, and imbalanced training data. Hence, we present \textbf{Two-Stage Mixture-of-LoRA}, a framework built on MedGemma-1.5-4B. The framework uses a shared-specific Mixture-of-LoRA architecture comprising one shared LoRA and six task-specific expert LoRAs, together with a two-stage training procedure. In Stage 1, we jointly train the shared LoRA and all task-specific expert LoRAs on all tasks. In Stage 2, we first freeze the backbone, the shared LoRA, and all non-target experts, and refine one task expert at a time. Classification and regression then receive an additional modality-balanced continuation, in which smaller modality groups are repeated to match the largest group. In the FLARE 2026 Task 3 test sets, the proposed method achieves 0.85 balanced accuracy for classification, 0.48 micro-F1 for multi-label classification, 0.79 detection F1, and 17.39 regression MAE. Code is available at https://github.com/YuanYL03/MICCAI-FLARE-2026-Challenge-Task3-2D.