Large language models struggle with consistent care decisions in cancer treatment
Patient-specific rubric-based evaluation of large language models across longitudinal cancer care
Artificial Intelligence
Summary
Evaluating advanced language AI models on cancer care shows that doing well in diagnosing doesn't mean they perform well in follow-up care steps. The authors created a new benchmark that tracks how decisions should change over multiple stages of cancer treatment using real patient records. They found that mistakes shift from wrong decisions to missing important steps as care progresses. Different prompt styles affect how good and safe the AI decision-making is. This research highlights the need for AI tools to adapt throughout the entire treatment process, not just at diagnosis.
What this means in practice
- •For medical ai developers: Evaluate AI models with phase-specific cancer care benchmarks to improve consistency across diagnosis and treatment phases.
- •For oncology informatics teams: Use patient-specific, rubric-based assessment tools to detect and mitigate different error types in multistage cancer care decision making.
Authors
Wei-Wei Zhang, Pengkai Wang, Yan Li, Zhijie Sang, Guan-Qun Zhou, Jun-Yan Li, Shu-Han Zhou, Pei-Wen Liu, Ting-Xi Tang, Min Tang, Zhitian Hou, Jianmin Wu, Jian Guan, i Lin, Hongxia Yang, Ying Sun
Abstract
Large language models (LLMs) show promise in specialty medicine, yet existing evaluations largely focus on isolated clinical encounters rather than tracking how clinical responsibilities evolve across a longitudinal care pathway. Here we introduce NPC-RubricBench, a phase-aware benchmark for evaluating multistage clinical reasoning across the cancer care continuum, using nasopharyngeal carcinoma as a defined disease context. Curated from longitudinal records of 405 patients across two cancer centers, the benchmark comprises 941 patient-phase units across six sequential care phases and 17,688 patient-specific atomic criteria spanning five evaluation dimensions, with each assessment strictly restricted to decision-time clinical evidence. Evaluating 30 general-purpose and medical LLMs revealed that strong diagnostic performance does not carry over to downstream care. We evaluated 30 general-purpose and medical LLMs, showing that high diagnostic performance did not translate into improved downstream care. Instead, the dominant failure modes reversed across phases. Explicit decision errors dominated anatomical staging at 48.2%, whereas decision omissions dominated initial work-up and post-treatment follow-up at 33.4% and 30.4%, respectively. In the evaluated model set, medically adapted models showed higher omission rates than general-purpose models. Moreover, varying query granularity exposes a critical specification trade-off: broad open-ended requests degrade Decision Quality across all 30 models, while more detailed, checklist-like prompts raise Decision Quality but reduce Evidence Faithfulness and Safety Adherence. Furthermore, NPC-RubricBench demonstrates that static diagnostic benchmarks are insufficient to ensure consistent clinical reliability over time, highlighting the importance of phase-adaptive safeguards in multistage medical AI.