Video generation models struggle to simulate medical procedures accurately
REMEDY: How Far Is Video Generation from Medical Education World Models?
Computer Vision and Pattern Recognition
Summary
Generating videos that show medical procedures convincingly is very hard because the videos must look real and also show the correct steps. The authors created a new test called REMEDY to check how well AI models can do this. They tested five different video generators on 12 medical tasks but found even the best model got less than 30% of the clinical steps right. This shows current technology is not yet good enough for producing reliable medical teaching videos.
What this means in practice
- •For medical simulation developers: Assess and improve video generation models for creating accurate medical teaching demonstrations.
- •For healthcare training content creators: Use the REMEDY benchmark to evaluate video realism and correctness before deploying AI-generated procedure videos.
Authors
Lixing Tan, Yanghao Zhou, Qing Xia, Yuting Guo, Shuai Li, Aimin Hao
Abstract
Recent video generation models produce realistic videos and show potential as a foundation for world models. These advances create opportunities for generating medical teaching demonstrations, which requires both convincing visual quality and precise procedural actions. However, whether current generators can meet these requirements has not been measured. To address this problem, we introduce Readiness Evaluation of Medical Education Demonstration sYnthesis (REMEDY), to our knowledge, the first benchmark for AI-generated medical teaching demonstrations. REMEDY provides 900 first frames from real demonstration videos, covering 12 tasks across four scenarios: operating room, imaging, clinic and bedside, and resuscitation. Five contemporary open-source video generation models produce 4,500 videos from these frames. We combine task-specific clinical checklists with video and motion quality metrics. Evaluation covers four dimensions: clinical action following, clinical profiles, video quality, and motion quality. Our results show that realistic appearance and temporal consistency do not ensure correct clinical actions. Even the most advanced MiniMax-H3 achieves only 28.25% on strict clinical success rate, and fine-grained clinical actions remain challenging. These findings establish a foundation and roadmap for developing future medical education world models.