MedHEB evaluates medical embeddings for cross modality evidence retrieval
MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence
Computer Vision and Pattern Recognition
Summary
Medical information comes in many forms like images, volumes, and text, which together help doctors understand diseases. The authors created MedHEB, a big test set with many different tasks, to see if current computer models can find related medical evidence across these different types. They found models work well in simple cases but have trouble linking evidence across different forms, like matching a text description to a specific image region. MedHEB also includes a baseline model and tools to help improve medical information retrieval for clinical use.
What this means in practice
- •For medical imaging developers: Benchmark and improve models that match medical images with related text and other images for diagnostic support.
- •For electronic health record teams: Enhance retrieval of relevant clinical reports and imaging studies across different modalities for better patient history access.
Authors
Yingshu Li, Shaoyang Zhou, Zhanyu Wang, Yunyi Liu, Xinyu Liang, Xi Zhang, Lingqiao Liu, Lei Wang, Luping Zhou
Abstract
Clinical evidence is inherently relational: medical images are interpreted with prior studies, cross-modal examinations, reports, and localized findings, and retrieving relevant evidence is therefore central to clinical interpretation. Recent foundation models increasingly encode these heterogeneous sources into shared embedding spaces, yet existing embedding evaluation often focuses on isolated matches, leaving unclear whether current models retrieve relevant evidence across changes in modality, appearance, and granularity. We introduce MedHEB, a unified benchmark with 107 retrieval tasks from 54 datasets, including both in-distribution and held-out-source evaluation, across three axes: evidence form (2D images, 3D volumes, text), pairing type (image--image, image--text, text--text, and modality-bridging visual matching), and relevance granularity (global, question-conditioned, region-level). Under a single ranking protocol, MedHEB varies clinically relevant evidence definitions to test whether embedding proximity remains meaningful across heterogeneous evidence, including modality-bridging visual retrieval and region-level text-to-region retrieval. Across representative models, performance is uneven across retrieval families: models strong on conventional image--text or text--text retrieval are less reliable on modality-bridging and region-level tasks. Current embedding spaces capture some clinical relationships well but do not yet provide balanced retrieval across heterogeneous evidence. We further provide MedEmb, an MLLM-based reference embedding model trained on the MedHEB ID split, as a reproducible baseline for unified 2D--3D--text medical retrieval. We release MedHEB with preprocessing scripts, evaluation code, access manifests, and MedEmb to support research on clinically meaningful medical embeddings.