Calibrated abstention improves reliability of large language models

The Missing "I Don't Know": Why Three Reasoning-Reliability Findings Converge on Calibrated Abstention

Machine LearningArtificial IntelligenceComputation and Language

Summary

Large language models often struggle with knowing when they don't know the answer, leading them to give wrong or misleading responses. The authors analyze three separate findings that each highlight a different problem caused by this lack of a built-in 'I don’t know' mechanism. They argue that the missing piece is a way for models to abstain from answering when uncertain, and that current evaluation methods don’t encourage teaching models this skill. They suggest new ways to measure model reliability that include tracking how often models decline to answer and how well they are calibrated.

What this means in practice

  • For ai system developers: Design evaluation frameworks that include abstention rates and calibration metrics to better measure and improve model honesty and reliability.
  • For software quality assurance teams: Integrate abstention-aware tests to detect and reduce incorrect model outputs in safety-critical AI systems.

A position paper. It proposes an approach and reports no results.

Authors

Srijith Ravikumar

Abstract

Three recent results describe what look like unrelated LLM reliability problems. Yin et al. (2026) show reasoning RL collapses tool-reliability representations. Suleymanov et al. (2026) show that under safety-constrained generation, large models rewrite flagged spans while small models truncate. Bastounis et al. (2024) prove any consistent-reasoning system without an implicit "I don't know" function must hallucinate infinitely often on broad problem classes. We argue these findings converge on a single intervention: calibrated abstention is what each independently identifies as the missing capability, even though the unavailability they document, a capability gap, a policy gap, and a recursion-theoretic gap, has a different source in each case. Honesty post-training has narrowed the gap in deployed models, but principled closure of the class Bastounis identifies requires a calibrated abstention function whose training signal at the leaderboard level is absent: dominant benchmarks assign zero reward to decline, so the leaderboard gradient that would select for the function does not exist. We propose four changes to evaluation: triple-scoring, abstention-rate reporting, capability-stratified evaluation, and mandatory calibration metrics. Benchmark reform is necessary, not sufficient, for closing the gap the theorem identifies.
View PDFOpen arXiv