Medical model outputs change when run on multiple devices
Reliability Testing of Medical Model Performance under Distributed Deployment
Artificial Intelligence
Summary
Medical AI models are usually tested on a single computer, but in real use they often run across many computers to work faster or handle bigger data. The authors show that when a medical model is run on multiple devices instead of one, its results can be different from what was expected. They built a test system and new benchmarks to check if the model behaves the same in both setups. Their experiments found that for many medical models, especially those that use images or multiple types of data, running the model distributed can cause noticeable differences in the results.
What this means in practice
- •For hospital data teams: Detect output inconsistencies when deploying medical AI models across multiple devices to ensure reliable patient diagnosis workflows.
- •For medical software developers: Use the distributed-execution-sensitive benchmark to verify medical AI model behavior matches centralized testing before deployment.
Authors
Yifei Wang, Xiaohan Zhang, Youtao Ding, Tianlin Li, Xiaoyu Zhang, Yida Yang, Li Pan
Abstract
Distributed inference has become an indispensable part of deploying medical models under practical latency, memory, and throughput constraints. Although modern frameworks improve serving efficiency through tensor parallelism, mixed precision, kernel fusion, and multi-device communication, they are generally assumed to preserve the behavior observed during centralized HuggingFace evaluation. This assumption creates an evaluation-deployment mismatch: a model may pass offline evaluation but produce a different output after the execution stack changes. To address this mismatch, we propose a testing framework and an improved, distributed-execution-sensitive medical-model benchmark that evaluates the same checkpoint and input under a centralized HuggingFace reference and matched distributed deployments. Extensive experiments across language, vision, and multimodal medical models show that execution changes can produce measurable output disagreements. Across supported visual settings, the test success rate ranges from 0.21 to 0.43 for single-modality models and from 0.32 to 0.98 for multimodal models. The benchmark is aimed at extending medical-model evaluation from capability and security to evaluation-deployment consistency.