Argument graph method estimates uncertainty from reasoning chains
Argument Graph Uncertainty: Quantifying Uncertainty from the Logical Structure of Reasoning Chains
Artificial Intelligence
Summary
Estimating uncertainty in AI answers is important, especially for tricky questions, but usual methods can be expensive or simplistic. The authors present a way to measure uncertainty by analyzing the logic structure within a single step-by-step reasoning process. They break down the reasoning into pieces, check how well these parts connect logically, and combine this into a confidence score. This approach works better than some existing methods and adds a fresh signal to improve uncertainty estimates without extra heavy computation.
What this means in practice
- •For ai system operators: Determine the reliability of answers from AI reasoning models to safely deploy them in critical decision-making workflows.
- •For software engineers in natural language processing: Integrate interpretable uncertainty estimates based on logical structure without costly model reruns for enhanced AI reasoning modules.
Authors
Simon Baur, Sumire Honda, Wojciech Samek, Jackie Ma
Abstract
Reliable uncertainty estimation is essential for deploying large language models in high-stakes reasoning tasks, yet most existing approaches rely on expensive sampling strategies or shallow token-level signals that ignore the internal logical structure of model outputs. We introduce Argument Graph Uncertainty, a post-hoc framework for multiple choice question tasks, that estimates model confidence directly from the logical structure of a single chain-of-thought reasoning trace. Our method uses either a lightweight instruction-tuned model or a simple deterministic rule to segment the reasoning chain into propositional units, then applies a NLI model to infer relational edges and construct two complementary argument graphs over these segments. The resulting edge scores are aggregated into a Dirichlet posterior over answer options, whose concentration yields calibrated uncertainty estimates reflecting both local step-to-step coherence and global logical consistency across the reasoning trace. Evaluated on GPQA, MMLU-Pro and TruthfulQA, our method consistently surpasses token-based and consistency-based uncertainty baselines across multiple reasoning models. We further demonstrate through extensive combination analyses that argument graph uncertainty captures an orthogonal signal to existing approaches, making it a complementary component in uncertainty ensembles. Crucially, our framework requires no additional calls to the evaluated model and relies entirely on small, efficient auxiliary models, producing fully interpretable, per-node uncertainty attributions at low computational cost.