Medical vision language models improve diagnosis by focusing on tough cases
Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Medical AI models often get better by practicing easy cases, but struggle with tricky diagnoses that need careful reasoning. The authors developed a two-step method called MedCBR to train these models to focus on difficult cases and follow detailed medical rules when uncertain. This approach helps the models make more accurate and well-explained diagnoses, especially for complex medical questions. Tests on six medical datasets show that MedCBR works better than previous methods for both broad and specialized tasks.
What this means in practice
- •For clinical ai developers: Improve medical imaging diagnosis systems by training them to better handle difficult cases and verify reasoning.
- •For healthcare software engineers: Build medical decision support tools that prioritize uncertain cases with clear clinical explanations for safer patient care.
Authors
Hongyu Bai, Jiaqi Zeng, Chunzheng Zhu
Abstract
Medical diagnostic reasoning requires verifiable links between visual evidence and clinical knowledge, yet current medical vision language models are aligned with objectives that reward the wrong behavior. Uniform supervised fine tuning lets easy cases dominate updates while clinically critical boundary cases saturate prematurely. Sparse rewards based only on final-answer correctness cannot assess anatomical grounding or clinical logic, inviting reward hacking through answer shortcuts. We propose MedCBR, a two-stage alignment framework coupling boundary-aware preference optimization with rubric-grounded reinforcement learning. First, we construct counterfactual boundary preferences at the model's capability frontier using empirical pass rates and preference pairs that share reasoning prefixes but diverge at decisive diagnostic branches. Difficulty Weighted Boundary Preference Optimization (DBPO) concentrates learning on this calibrated boundary region. Second, we distill reusable clinical rubrics from boundary trajectories and activate them according to prediction entropy. Entropy Gated Grounded Policy Optimization (EGPO) rewards answer correctness, anatomical grounding, and reasoning fidelity while suppressing rubric noise on reliable samples. A reference policy anchored to the boundary-aligned model couples population-level boundary priors with instance-level uncertainty, making diagnostic reasoning more accurate and clinically grounded. Experiments on six medical benchmarks show consistent improvements over strong general, specialized, and reinforcement learning baselines, reaching 61.2% average accuracy with the largest gains on challenging differential diagnosis tasks. Ablations show super-additive gains: combining the two stages exceeds the sum of their individual improvements.