Medical model improves image questions and segmentation separately
SAM Meets VLM: Parameter-Decoupled Full-Parameter Training for Unified Medical Reasoning and Segmentation
Computer Vision and Pattern Recognition
Summary
Medical language-and-vision models struggle to both understand clinical questions and highlight exact areas in images related to their answers. The researchers created a training method that separates parts of the model responsible for text reasoning from those handling image segmentation, so they don’t interfere with each other. This approach helps the model better find and outline medical features in images while still answering questions accurately. Their tests show the model balances both skills effectively.
What this means in practice
- •For medical imaging software teams: Improve clinical image analysis tools with models that better connect questions and exact image areas for diagnosis support.
- •For radiology workflow managers: Integrate accurate image segmentation with language-based clinical queries to speed up radiology report generation and review.
Authors
Xuyang Cao, Enyou Liu, Jun Zhao, Zhuoyun Liu, Jintao Fei, Leo
Abstract
Medical multimodal large language models (MLLMs) are increasingly expected not only to answer clinical questions, but also to localize the visual evidence behind their predictions. A common strategy connects a vision--language model (VLM) with SAM-style segmentation through a special <SEG> token, yet full-parameter training of this unified architecture is difficult because image-level reasoning and pixel-level segmentation impose different requirements on the shared representation space. To address this issue, we propose a parameter-decoupled training framework for unified medical reasoning and segmentation. The framework treats the <SEG> hidden state as a semantic-to-spatial prompt for the mask decoder and encourages it to become separable from generic language states, reducing ambiguous segmentation prompts and potential disruption to reasoning representations. It first performs medical shallow alignment to adapt visual features to clinical language without disturbing the LLM; then controlled instruction tuning shapes separable <SEG> prompt states, monitored by the Davies--Bouldin Index (DBI), while scaling segmentation gradients entering the language backbone; finally, the SAM branch is specialized with the VLM frozen to improve mask precision without altering reasoning parameters. Experiments on medical referring segmentation, grounding, visual QA, and textual QA benchmarks show that our framework achieves strong language-conditioned segmentation while preserving competitive reasoning ability. Ablations show that two-phase instruction tuning, gradient scaling, and segmentation specialization all contribute to the model.