Confidence reasoning graphs improve trust in large language model agents
Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents
Artificial IntelligenceComputation and Language
Summary
When large language model (LLM) agents solve complex tasks, it's hard to know how much to trust their answers. The authors propose a method called Confidence Reasoning Graphs (CRGs) that breaks down the agent’s reasoning steps and estimates confidence for each part. By combining these estimates, CRGs give a more reliable overall confidence score without needing extra training data or internal signals from the model. This helps users decide when to trust the agent's output or intervene.
What this means in practice
- •For ai system operators: Estimate confidence in LLM agent task success to decide when human review or intervention is needed.
- •For automated customer service teams: Improve reliability by monitoring confidence in generated responses to reduce errors without costly retraining.
Authors
Brendan King, Farima Fatahi Bayat, Jean-Flavien Bussotti, Pouya Pezeshkpour, Estevam Hruschka
Abstract
When using an LLM agent in a consequential domain, making an informed decision about whether to trust its output or intervene requires calibrated confidence in the agent's success. Confidence estimation for agents is difficult because evidence about success is distributed across heterogeneous, interdependent steps of an agent's trajectory. Practical agentic deployments introduce further challenges: frontier LLMs often provide limited access to internal signals, agent roll-outs are costly, and training data may be unavailable or quickly become outdated. To address these challenges, we introduce Confidence Reasoning Graphs (CRGs), an inference-time framework that estimates the probability an agent accomplished its task from a single trajectory, without privileged model access or training data. Rather than compressing an execution into a single holistic judgment, a CRG begins with the claim that the agent accomplished its task, decomposes it into contextualized sub-claims grounded in trajectory evidence, estimates confidence for each terminal claim, and finally aggregates these into an overall confidence estimate. Across three agentic benchmarks, three backbone models, and three agent frameworks, CRGs yield better-calibrated confidence and stronger risk-aware decision making than verbalized, sampling-based, and white-box surrogate baselines. We further find that calibration error alone can be misleading: a white-box surrogate baseline appears well calibrated while providing near-chance discrimination. Ablations attribute CRG's improvements to claim-level confidence estimation and aggregation rather than graph construction alone. Finally, a CRG exposes the claims and trajectory evidence underlying each confidence estimate, enabling it to be audited at decision time.