Evaluating and Understanding Model Editing for Medical Vision Language Models
2026-07-06 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created M3Bench, a new test to check how well medical vision-language AI models can be fixed after deployment without full retraining. Their benchmark mimics real clinical challenges like changing images, different medical protocols, and evolving patient data. By testing four fixing methods on six models, they found none worked perfectly across all real-world demands; some methods handled changes well but caused errors nearby, others kept local fixes but struggled with generalizing. They linked these issues to how the models organize information internally. Overall, M3Bench helps researchers better understand and improve safe updates to medical AI systems.
Vision-language modelsModel editingMedical imagingBenchmarkClinical domainMultimodal learningGradient-based methodsMemory-based methodsLatent spacePost-deployment adaptation
Authors
Guli Zhu, Chenwei Wu, Liyue Shen
Abstract
Model editing promises a fast, targeted way to correct post-deployment mistakes in medical vision-language models (VLMs) without costly retraining. However, existing multimodal model editing benchmarks focus on general-purpose tasks and do not reflect realistic clinical domain requirements and variability. To address this, we introduce M3Bench, a clinically grounded benchmark for multimodal model editing that evaluates whether an edit remains reliable, precise, and generalizable under the challenges of image and text variation, modality and protocol shifts, clinical knowledge composition, and temporal progression. M3Bench contains 16,276 questions spanning diverse anatomy, modalities, and specialties, and supports both single and sequential edits. By evaluating 4 representative editors across 6 medical and general VLMs, we find that no method excels across all criteria. Gradient-based editors achieve strong transfer but suffer from catastrophic locality violations, whereas memory-based methods preserve locality but lack compositional generality and exhibit high backbone-dependent hyperparameter sensitivity. We further attribute these failures to the latent space geometry of VLMs and how different editing methods shift its landscape. Overall, M3Bench establishes a rigorous clinical stress test for multimodal model editing and offers actionable guidance for safer post-deployment adaptation. The benchmark is publicly available at https://github.com/BioMed-AI-Lab-U-Michgan/M3Bench .