Language model reliability improved by reasoning aware calibration
Reasoning-Aware Multicalibrated Reliability Predictors for Language Models
Computers and SocietyArtificial Intelligence
Summary
People rely on language AI models for answers, but sometimes the models make mistakes called hallucinations. The authors found that checking the model’s reasoning steps (chain-of-thought) alongside the topic it’s answering about can better predict when answers are reliable. They created a method that groups answers not just by topic but also by reasoning traits to improve trustworthiness predictions. Tests showed this method reduced error in spotting unreliable or unsafe responses better than looking at topic alone.
What this means in practice
- •For ai product developers: Build better AI assistants that can more accurately signal when their answers may be incorrect by monitoring reasoning patterns alongside topics.$Commercial implications: Enables development of AI software products with improved answer reliability warnings, increasing user trust and satisfaction.
- •For content moderation teams: Use reasoning-aware reliability predictors to identify and reduce unsafe responses generated by language models on various topics.
Authors
Ekaterina Fedorova, Alessandro Morosini, Chara Podimata
Abstract
As people increasingly rely on large language models (LLMs) for information, model providers need to communicate meaningful estimates of whether individual responses are correct or, more generally, reliable. In the context of hallucinations, existing work uses multicalibration to make such reliability predictions meaningful across application-relevant groups, such as prompt topics. Topic groups alone, however, can conceal systematic errors: within the same topic and confidence level, responses with different chain-of-thought (CoT) properties may exhibit opposing errors that cancel on average. In this work, we introduce reasoning-aware multicalibration, a framework that supplements application-relevant topic groups with groups derived from observable properties of CoT traces. We highlight that, theoretically, groupings based on CoT are "internally important", i.e., groupings for which the multicalibration guarantee is not societally crucial as it is for high-risk topics, but whose inclusion in the multicalibration procedure could easily and significantly improve predictive accuracy of the final reliability predictor. Empirically, we apply our framework to hallucination prediction and unsafe-response detection across a variety of benchmarks, and find that adding CoT groups to topic-based calibration reduces Brier score in both tasks. Interestingly, we find that multicalibrating with respect to CoT groups alone (without any topic information) already improves calibration across topics, suggesting that the predictor's errors depend on how the model reasoned, rather than what it was asked about in the prompt.