Medical vision language model post-training shows mixed image use
From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Medical vision-language models are tested by how well they answer questions about images, but this study looks deeper at how training changes their use of images for decisions. The authors compare different training methods and find that improving overall accuracy doesn’t always mean the model relies more on the image. Some training makes the model less sensitive to images, while others create unstable outcomes. Their detailed analysis shows a gap between how these models are optimized and how they actually use visual information in practice.
What this means in practice
- •For medical ai developers: Improve evaluation methods for medical vision-language models by auditing how training affects their use of images in decision-making.
- •For clinical data teams: Assess and select better post-training strategies for medical visual question answering systems based on true image utility rather than accuracy alone.
Tested on one dataset.
Authors
Wang Jingxin
Abstract
Medical vision-language model (VLM) post-training is commonly evaluated through answer accuracy. We examine how changes in accuracy and training objectives relate to image-conditioned decisions in a controlled Qwen2.5-VL-3B study on PMC-VQA. We compare supervised fine-tuning (SFT) with low-rank adaptation (LoRA) restricted to the language model, expanded multimodal adaptation scopes, standard answer-only Group Relative Policy Optimization (GRPO), and a counterfactual evidence objective. On 2,000 clean-test questions, language model LoRA SFT changes correct-image accuracy by +1.10 percentage points (95% paired bootstrap CI:-0.85 to +3.05), while visual-benefit events decrease by 2.40 points and image sensitivity decreases by 5.60 points. Paired records reveal 155 acquired and 203 lost visual-benefit events. Broader adaptation yields lower correct-image accuracy than language-model LoRA SFT. Standard GRPO produces mixed-reward groups and parameter updates, with an uncertain clean test accuracy change. A generation audit reveals that canonical option scores can follow a different token path from generated answers. With scores taken along the greedy generation path, the evidence target improves on the training set; its gains over standard GRPO remain inconsistent on validation data at matched training doses. Sample-level analyses trace how evidence scores, decision margins, and generated answers change during post-training. This empirical and measurement audit identifies gaps between optimization activity, target acquisition, and useful held-out visual behavior.