GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created GPAgentBench-2K, a new test for large language models (LLMs) that helps them make decisions like real-world primary care doctors. Unlike earlier tests, this one includes many types of clinical actions and considers safety rules, like when the model should choose not to act. When testing 16 top LLMs, the authors found that as the decision options increased, performance dropped and safety was often compromised. Even the best models made unsafe choices in more than half of the risky situations. They also showed that adding safety rules to the training helps but is still not good enough for real clinical use.
Large Language ModelsConstrained Markov Decision ProcessClinical Decision-MakingPrimary CareSafety ConstraintsReinforcement LearningPolicy OptimizationClinical WorkflowsAbstentionBenchmarking
Authors
Boqi Chen, Xudong Liu, Yunke Ao, Heejin Do, Jianing Qiu
Abstract
Large Language Models (LLMs) show great potential as clinical agents, yet existing benchmarks reduce clinical workflows to static predictions or unconstrained Markov Decision Processes (MDPs) with coarse action sets. To address this, we introduce GPAgentBench-2K, the first Constrained MDP (CMDP) LLM-agent benchmark for primary-care clinical decision-making, constructed from expert-validated records of real-world GP encounters. Our environment models a full spectrum of six foundational clinical actions, imposes a topological workflow prior over the action space, and operationalizes safety-informed abstention as a first-class outcome. Evaluating 16 state-of-the-art LLMs reveals a significant performance degradation as the action space scales. Crucially, we uncover a clinical quality-safety gap: even frontier models with the highest diagnosis accuracy violate safety constraints in over half of high-risk cases. Finally, we establish a reference point using Constrained Group Relative Policy Optimization (C-GRPO), and show that while explicitly modeling constraints improves performance over unconstrained RL methods, it remains far from clinically acceptable safety.