Vision language model improves continuous measurement accuracy in medical images
Beyond Discrete Tokens: Preserving Numerical Continuity in Medical Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Measuring things in medical images, like ultrasound scans, is hard because numbers are continuous but computers often predict them as separate words or tokens. The authors created a new model called HAD-VLM that mixes language prediction with a method that handles continuous numbers better. This model measures parts of the heart in ultrasound images more accurately and can also answer medical questions and measure bones from X-rays. It works better than older models that only used word-based predictions.
What this means in practice
- •For medical imaging software developers: Integrate HAD-VLM to enable prompt-based, accurate continuous measurements in ultrasound and X-ray analysis tools.$Commercial implications: Enables advanced medical imaging software products with improved measurement accuracy and language interaction capabilities for hospitals and diagnostics companies.
- •For healthcare ai operations teams: Use HAD-VLM models to reduce errors in automated echocardiographic and vertebral measurements, improving clinical workflow efficiency.
Authors
Dianye Huang, Jiamei Guo, Yue Zhou, Mingde Li, Yi Xiong, Wei Wang, Nassir Navab, Zhongliang Jiang
Abstract
Objective: To develop a unified vision-language model for prompt-conditioned quantitative ultrasound measurement that addresses the mismatch between continuous measurement targets and discrete token-based numerical prediction. Methods: This article presents HAD-VLM, a hybrid autoregressive-diffusion framework that combines autoregressive text generation with continuous measurement prediction within a shared backbone. Visual prompt tuning adapts the vision encoder to ultrasound images. A diffusion process predicts a five-component representation comprising caliper endpoint coordinates and measurement length. A post-processing strategy combining the proposed coordinate-length consistency mechanism with ensemble prediction is introduced to refine measurement estimates. We evaluate four echocardiographic measurement tasks on EchoNet-LVH, with additional assessments of medical question answering and X-ray vertebral measurement. Results: HAD-VLM achieves mean absolute errors of 1.36, 2.77, 3.05, and 1.16 mm for IVSd, LVIDd, LVIDs, and LVPWd, respectively, reducing errors by 21.4-37.6\% relative to the Vicuna-based autoregressive baseline. It also outperforms fine-tuned Qwen3-VL and direct MLP regression across all four tasks. Medical question answering performance is preserved on the evaluated dataset. For X-ray vertebral measurement, HAD-VLM reduces mean absolute error from 11.4 to 6.7 pixels. Conclusion: Integrating autoregressive language generation with diffusion-based continuous prediction improves quantitative medical image measurement while retaining prompt-conditioned task flexibility. Significance: HAD-VLM provides a unified approach to combining language interaction with anatomically grounded numerical prediction for quantitative medical image analysis.