Clinical ai agents show different orders on identical patient cases
Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs
Computation and LanguageArtificial IntelligenceMachine LearningSoftware Engineering
Summary
Clinical AI agents meant to help doctors sometimes give different medical orders for the exact same patient information when run multiple times. The researchers ran tests replaying identical cases through these agents and found that their suggested tests, medications, and referrals often changed between runs, even though the overall benchmark scores stayed the same. This means the systems can behave inconsistently in practice, which could affect patient care. The authors suggest evaluating these agents multiple times and checking the consistency of their specific actions, not just their overall scores.
What this means in practice
- •For clinical ai developers: Evaluate and improve AI agent consistency by comparing repeated runs on identical medical cases.
- •For hospital data teams: Monitor AI clinical tool reliability by assessing repeated-action stability to ensure consistent patient recommendations.
Authors
Rohith Reddy Bellibatlu, Manpreet Singh, Zhoutian Han, Wenbin Zhang
Abstract
A clinical agent benchmark can report the same verdict on identical inputs while the agent files a materially different order on each run. Such agents order tests, request medications and place referrals, yet benchmarks typically score one run per task and rarely ask whether identical inputs produce identical actions; MedAgentBench, the benchmark we use, scores a single attempt and says so. To measure this gap we introduce "same-input rerun", which replays a task with every input held fixed and compares the orders rather than the score, with six reliability metrics, and apply it to 1000 MedAgentBench runs across 50 tasks from its five write-capable families, two open-weight models below ten billion parameters quantised to four bits, and two temperatures. The study establishes that action-level divergence exists and can pass unrecorded by the score, not that any rate generalises. Under the 8B model at temperature 0.7, all 43 ordering groups emit a different set of orders across five identical runs, 26 emit the order on some runs and not others, and 28 record a different coded value, dose or analyte. In 22 of those 43 the benchmark reports the same failing verdict for materially different behaviour, as it does for all 10 divergent groups of the 4B model at 0.7. Orders also reach different endpoints across runs, one of which the record server rejects while the agent is told it succeeded. These findings motivate repeated-run evaluation, action-level stability reporting and execution-faithful environment feedback in clinical-agent benchmarks.