MedBenchAgent automates building medical imaging model benchmarks reliably
MedBenchAgent: Towards Systematic Automation of Medical VLM Benchmark Construction
Artificial Intelligence
Summary
Medical vision-language models need special tests to see how well they understand images and text related to healthcare. The authors created MedBenchAgent, a system that automatically designs these tests by figuring out what to check and using medical knowledge and data. This system carefully plans and then builds test questions, achieving high accuracy verified by humans. It also works well in specific medical fields and helps compare different medical AI models more clearly.
What this means in practice
- •For hospital data teams: Automatically generate and verify tests to evaluate medical AI models using existing annotations and medical knowledge.
- •For medical ai engineers: Create detailed benchmarks that reflect task-specific challenges for better tuning and comparing medical vision-language models.
Authors
Yulin Fu, Junren Wang, Guangjing Yang, Zhangyuan Yu, Wanran Sun, Jiabao Zhou, Jin Yin, Qicheng Lao
Abstract
Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large language models (LLMs), yet existing automation largely focuses on generating evaluation items within predefined benchmark specifications. We study the broader problem of automatically deriving the specification itself: what to evaluate, which annotations support each task, and how to translate this evidence into reliable evaluation items. We formulate benchmark construction as constrained compilation, in which the benchmark specification is progressively derived from evaluation requirements, heterogeneous annotations, and medical knowledge. Based on this formulation, we introduce MedBenchAgent, a multi-agent framework with a Benchmark Intermediate Representation (BIR) that encodes task definitions, evidence mappings, evaluation protocols, and item specifications across construction stages. MedBenchAgent separates planning, which derives and verifies the specification, from instantiation, which constructs and audits items under the locked specification. MedBenchAgent achieves a Task-Space F1 of 90.9%, outperforming direct task induction (79.2-80.0%) and prior-guided induction (85.1%); 994 of 1,000 sampled items from correctly identified tasks pass human audit. We further demonstrate portability to a specialized medical domain and evaluate twelve VLMs, revealing task- and setting-specific variation obscured by aggregate scores. These results establish constrained compilation as a scalable and auditable framework for medical VLM benchmark construction beyond question generation.