Open large language models struggle to align clinical risk and cost choices
Misaligned Clinical Risk Classification and Cost Asymmetry in Open-Weight Large Language Models
Machine LearningArtificial Intelligence
Summary
Large language models can predict patient risk levels about as well as traditional tools. However, this study found that these models don’t reliably adjust their decisions based on the cost of making different errors, like missing a diagnosis or making a false alarm. The authors tested several models on diabetes data and saw that only the biggest models somewhat reflected cost directions, but none fully handled how much those costs mattered. This means these AI models might need extra checks to make sure they consider both patient risk and the cost of mistakes before being used in healthcare.
What this means in practice
- •For hospital data teams: Assess AI risk prediction tools for how well they handle clinical cost tradeoffs before deployment.
- •For medical ai developers: Design clinical language models to better integrate cost asymmetry into decision outputs for safer recommendations.
Tested on one dataset.
Authors
Star S. D. Liu, Xiyu Ding, Robert B. Barrett, Alberto Santamaria-Pang, Nic Dobbins, Harold P. Lehmann
Abstract
How large language models (LLMs) integrate patient risk with clinical cost tradeoffs remains poorly understood. We investigated how four open-weight LLMs (Qwen-2.5-7B/32B and Llama-3.1-8B/70B) internally represent cost tradeoffs, how these representations relate to clinical predictions, and whether decisions shift as predicted by the specified cost direction and magnitude. Using a public diabetes dataset, we varied 11 false-negative (FN) to false-positive (FP) cost ratios across three phrasings and examined representations and behavioral outputs. Patient risk was linearly recoverable on par with conventional classifiers (AUC $\approx 0.83$), and cost direction was recoverable in every model. However, representational shifts in cost direction tracked output changes only in the two larger models, and responses to cost magnitude were predominantly direction-agnostic. Only 2 of 12 model-phrasings showed both opposing responses to increasing FN versus FP costs and cost-correct ordering. Representationally, a direction fitted on one cost side did not invert when transferred to the other, as expected under mirror-symmetric encoding. These findings suggest that LLMs encode risk and cost information but do not reliably integrate them into cost-correct decisions. Clinical evaluations should therefore include tradeoff tests, phrasing sensitivity, and default operating points alongside predictive performance.