Benchmark tests ai control on medical imaging machine interfaces
MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Medical imaging machines like CT and ultrasound need careful control to get good pictures of the body. The authors created MedImageOSWorld, a computer simulation that tests how well AI agents can handle these machines by using the same screen controls humans do. They found that while some AI can follow many steps of the machine operation, they often fail to capture the exact anatomy intended. The test helps show where AI succeeds or struggles in managing medical imaging tasks.
What this means in practice
- •For medical device developers: Evaluate and improve AI-driven control software for medical imaging consoles in realistic simulated settings before clinical use.
- •For healthcare it teams: Test AI tools that assist technicians in optimizing imaging protocols to enhance workflow efficiency and image quality.
Authors
Ziyang Long, Xinqi Li, Lujing Xing, Hsin-Jung Yang
Abstract
Graphical consoles offer a practical interface for medical acquisition assistance, allowing agents to work through the controls and visual feedback used by human operators. Reliable assistance requires linking on-screen anatomy to acquisition decisions that determine what image evidence becomes available next. We introduce MedImageOSWorld, a benchmark for evaluating this capability in simulated CT, MR, and ultrasound consoles. Using screenshots and mouse-and-keyboard actions, agents configure protocols, plan acquisitions, inspect the resulting images, and make corrective adjustments across seven capability levels, from console operation to feedback-driven control. Evaluation combines task-specific workflow checks with hidden anatomical ground truth to assess procedural completion and acquisition outcomes separately. A common evaluation protocol specifies episode conditions and interaction budgets, while recorded trajectories support analysis of how agents observe, act, and respond to acquisition feedback. Across eleven open-weight agents, success rates range from 3.0 to 25.0 on a 0-100 scale while workflow-progress rates reach 21.5-74.5: agents complete much of the console workflow but rarely acquire the intended anatomy. Success collapses between perception and planning, from 79-90% at the lowest three levels for the best agent to at most 12% for millimetre-level planning and 0% for closed-loop control among open-weight agents. Two proprietary agents reach success rates of about 34 and exceed the best open-weight agent mainly in acquisition quality (61 versus 42). MedImageOSWorld provides a controlled setting for studying whether general-purpose GUI agents can translate visual observations into effective medical acquisition decisions.