UltraG-Bench assesses pixel-level ultrasound image understanding

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Ultrasound is a common medical imaging method, but current AI models struggle to point out exactly which parts of an image relate to their descriptions. The authors created UltraG-Bench, a big benchmark with many annotated ultrasound images to test how well models can link their answers to precise spots in the images. They tested 14 top AI models and found most still miss fine details even if they get the general idea right. To improve this, the authors made UltraG-Agent, a system combining language reasoning with specialized ultrasound image segmentation, which showed better results in matching image details with explanations.

What this means in practice

  • For medical imaging engineers: Use UltraG-Bench to evaluate and improve ultrasound AI models’ ability to localize image evidence supporting their findings.
  • For clinical ai software developers: Apply UltraG-Agent to enhance medical ultrasound applications with better explanations linked to precise image regions.$Commercial implications: This enables ultrasound diagnostic tools with interpretable AI features that can be marketed to healthcare providers.

Authors

Quanhao Zhu, Bo Xu, Rui Lin, Chenyuan Wang, Yu Shao, Boling Zhu, Jiuyan Sun, Liang Zhao, Hongfei Lin, Feng Xia

Abstract

Ultrasound is one of the most widely used medical imaging modalities, and recent large vision-language models(VLMs) have shown increasing capabilities in ultrasound image understanding. However, these models fail to provide pixel-level visual evidence aligned with their semantic predictions, and their fine-grained grounding capability in ultrasound remains largely unclear. We introduce UltraG-Bench, a large-scale multi-task benchmark for evaluating pixel-level evidence grounding in ultrasound. UltraG-Bench is built by annotating 40 public ultrasound segmentation datasets spanning 13 anatomical categories, and comprises three progressive tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation, with 331125, 666779, and 138832 annotations, respectively. Comprehensive evaluation of 14 state-of-the-art models reveals a substantial gap between semantic understanding and fine-grained pixel-level localization. We further propose UltraG-Agent, which combines the semantic reasoning capabilities of a VLM with the ultrasound-specific segmentation capability of UltraSAM3. Experiments show that UltraG-Agent substantially improves both semantic prediction and pixel-level visual grounding. Our dataset and code are available at https://github.com/zhuqh19/UltraG-Bench.
View PDFOpen arXiv