Language model confidence should be standard in evaluation
Calibration as a First-Class Criterion in LLM Evaluation
Computation and Language
Summary
Language models sometimes say how sure they are about their answers, but often people don't check if those confidence scores are accurate. The authors explain that this can cause problems both when using the models for real tasks and when researchers build tools that rely on these confidence scores. Because most evaluation tests already have the information needed to check confidence, the authors suggest always including confidence calibration as part of model evaluation. They also point out that for open-ended tasks, measuring confidence is still tricky and needs further work.
What this means in practice
- •For machine learning engineers: Include confidence calibration scores when assessing language models to improve trustworthiness in deployed applications.
- •For ai product teams: Ensure models used for synthetic data and automated judging are calibrated so their confidence reflects true accuracy.
A position paper. It proposes an approach and reports no results.
Authors
Mario Sanz-Guerrero, Katharina von der Wense
Abstract
Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is a well-studied subfield within NLP. Methods to measure it already exist. The problem is adoption: outside this subfield, NLP research regularly introduces new models, datasets, and benchmarks without checking whether the model's confidence scores are meaningful. We argue that this adoption gap is a major obstacle to trustworthy LLM evaluation. Miscalibration causes problems in two distinct areas: at deployment, where overconfident mistakes cause real harm, and inside the research pipeline, where methods like LLM-as-a-judge, synthetic data generation, and active learning rely on calibrated confidence without verifying it. Standard calibration metrics only require two inputs per example: a confidence score and a correctness judgment. Most benchmarks in use today already provide both, meaning calibration can be reported immediately. For open-ended generation, however, defining these two inputs is still an open challenge. We argue that each NLP subfield should pair its main performance metric with a calibration score and call for treating calibration as an essential property of every model rather than a niche topic.