Utilitybench measures language model errors by their real costs
UtilityBench: Measuring Language Model Utility When Errors Have Different Costs
Machine LearningArtificial Intelligence
Summary
Language models make mistakes, but not all mistakes are equally serious—some can have much bigger consequences than others. The paper presents UtilityBench, a new way to test language models by giving different weights to errors based on how bad they are in real life. They also created a new score called UtilityAUC that better reflects how well a model balances answering questions and avoiding costly errors. Their tests showed that being more accurate doesn't always mean a model makes the safest or most responsible choices.
What this means in practice
- •For healthcare ai teams: Evaluate and deploy language models that minimize harmful errors by incorporating cost-based utility metrics for safer clinical decision support.
- •For customer support teams: Prioritize when to have models respond or abstain based on the cost of errors to reduce serious mistakes in automated communication.
Authors
Qifeng Lei, Zhiyong Yang, Zitai Wang, Shilong Bao, Linchao Zhang, Qianqian Xu, Qingming Huang
Abstract
As language models are increasingly deployed in real-world scenarios with important outcomes, the impact of their mistakes often extends far beyond what correctness can capture. In domains like medicine, a single error can lead to serious consequences and put patient safety at risk. However, mainstream evaluation protocols still treat all errors as equal, overlooking the spectrum of real-world costs. To address this, we propose a utility-based evaluation paradigm that explicitly incorporates the varying consequences of errors. Guided by this principle, we develop UtilityBench, a large-scale benchmark of 15,000 instances with verifiable answers, quantifying error severity using Best-Worst Scaling to elicit meaningful, data-driven cost estimates. These quantified utilities shape when a model should answer or abstain, guiding it to make decisions that align with the true importance of each scenario. As another merit, we introduce UtilityAUC, a metric that measures the area under the utility curve determined by a model's response ordering, thereby enabling utility preference alignment beyond error frequency. Our theoretical analysis demonstrates that UtilityAUC can distinguish between models where standard accuracy and AURC fall short. Empirically, evaluating 20 state-of-the-art models, we find that greater accuracy or more advanced architectures do not reliably translate to higher UtilityAUC, highlighting a gap between problem-solving ability and responsible decision-making in language model evaluation.