Multimodal medical model improves data selection with visual necessity scores
ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection
Machine LearningArtificial Intelligence
Summary
Medical AI models that read images and text together can sometimes miss important visual details because they rely too much on text. The authors found that this happens because the models focus more on language and less on images when learning. They created a new method called ViCoR that picks out training data where the images are truly needed by looking at differences when the image is removed. This helps the AI learn better from visuals and improves its performance on medical tasks.
What this means in practice
- •For medical ai developers: Select more visually informative patient cases to fine-tune AI models that interpret medical images and reports together for better diagnosis accuracy.
- •For clinical data teams: Build smaller, more effective training datasets focused on key image-text examples to improve medical AI reliability and robustness across diseases.
Authors
Haotian Gan, Yudong Li, Xintian Li, Jiarui Han, Weidong Tang
Abstract
Medical vision-language models often exhibit hidden stratification, where high overall performance masks failures on subtle visual findings and overlapping clinical conditions. We trace this failure mode to modality imbalance during fine-tuning, where strong linguistic priors can suppress visually grounded learning and induce plausible but weakly grounded rationales. Existing data selection methods usually score image-text pairs as holistic units, without separating visually necessary samples from shortcut-solvable ones. To address this problem, we propose ViCoR, a counterfactual-residual-based data selection method for Med-VLM fine-tuning. ViCoR estimates visual necessity through image-dependent optimization residuals, calibrates the resulting scores with medical supervision compatibility, and constructs a diversity-aware fine-tuning subset. Across two backbones, four external generalization benchmarks, and one multi-disease robustness benchmark, ViCoR-selected 20\% subsets achieve the strongest average performance compared with full-data fine-tuning and selection baselines. Gradient-level analysis and perturbation audits further show that ViCoR enriches image-dependent training signals and strengthens image-level and evidence-region dependence. The code will be available at https://github.com/htgan-ai/ViCoR.