Healthcare AI models struggle to reliably escalate uncertain cases
Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures
Artificial IntelligenceComputation and Language
Summary
AI systems for healthcare must know when to ask for help or pass problems to a human, especially when information is missing or conflicting. This paper studies how well large AI language models handle these situations using a realistic set of medical workflows. The authors found that even high-performing models often miss important cases where escalation is needed, and their confidence measures are unreliable. Adding a transparent controller that tracks workflow steps helps reduce missed escalations but struggles with wording variations not seen before.
What this means in practice
- •For healthcare ai developers: Improve AI-based clinical support tools by auditing and controlling escalation decisions under uncertain patient data.
- •For ai safety teams: Design workflow-aware monitoring systems to reduce missed critical alerts in automated decision agents.
Authors
Zihao Zheng, Junyi Yao, Baichuan Li, Jiayu Long
Abstract
Trustworthy biomedical AI requires reliable action selection when evidence is missing, stale, conflicting, or late, not only accurate answers on static inputs. We instantiate an episode-level evaluation protocol as 400 synthetic healthcare workflow episodes: 40 hand-authored semantic seeds across four task classes, two surface parameterizations, and five controlled variants, scored on ANSWER/ASK CLARIFICATION/ABSTAIN/ESCALATE under a pre-declared 5:1 missed-versus-unnecessary escalation cost. Reference actions are single-author study-policy labels, not clinician-adjudicated ground truth. Across five open-weight configurations (1.7B--9.7B), clean decision accuracy reaches 0.97, yet uncertainty-prompted models miss 32--97% of required escalations. The stronger 9.7B configuration narrows but does not remove the gap, falling from 0.97 clean accuracy to 0.78 under conflict. Verbalized confidence is inconsistent (ECE 0.10--0.54; failure AUROC 0.48--0.77), and a development-tuned confidence-only policy yields no utility gain. A transparent workflow-aware controller improves 5:1 utility for every model and reduces missed escalations to 0.12--0.24, at specificity 0.83--0.93. However, a 30-concept lexical stress test reduces each rule detector's recall from 1.00 on trigger phrases to 0.00 on unseen paraphrases, limiting the controller claim to this benchmark. Seed-clustered statistics, ablations, cross-task transfer, raw outputs, and manifests are packaged for audit. The results motivate explicit workflow-state evaluation while showing that apparently effective routing controls need their own generalization tests.