Understanding when large language model agents can be trusted
Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study
Information Retrieval
Summary
Large language model agents are AI systems that can hold long conversations, make plans, and use tools, but it's hard to know when to trust them. The paper explains that uncertainty in these agents comes from many steps in their process, not just from single questions. The authors created a way to measure uncertainty more carefully by looking at different points in the agent’s reasoning journey. They found that just one confidence score doesn’t tell the whole story and that confidence can change as the agent goes through different stages. This helps make AI agents safer by showing where and when they might be wrong.
large language modelsagent systemsuncertainty quantificationmulti-turn conversationcalibrationtrajectory-level analysisexpected calibration errorconfidence estimationerror propagationAI safety
Authors
Moule Lin, Qizhen Lan, Shuhao Guan, Weipeng Jing, Jiexin Fan, David Gregg, Goetz Botterweck
Abstract
Large language models (LLMs) are no longer deployed only for single-turn conversation but increasingly act as agents that plan, call tools, retrieve evidence, maintain memory, and interact over long horizons, often together with other agents through multi-turn conversations. Therefore, knowing when to trust the agentic system is a prerequisite for safe deployment. However, existing work on quantifying uncertainty for LLMs was built almost entirely for single-turn question answering. This paper argues that errors and uncertainty arise from multi-turn conversations, environments, and tools rather than from a single-turn question answering setting. It comes late, however, and is compounded in a single score that is too coarse to represent the unreliability. We organize the literature with a three-axis taxonomy, (1) what the uncertainty is, (2) how it is estimated, and (3) where uncertainty arises during an agent pipeline. We investigate step-level and trajectory-level calibration and show with a simple counterexample that the first does not imply the second. Experiments on real agent traces across four models and up to a 50-step budget show that the proposed metric and reporting protocol (Trajectory-Checkpoint Expected Calibration Error, TC-ECE) can be computed and that step errors are coupled along a trajectory. We find that confidence estimates from the agent's own responses do not consistently outperform a simple baseline. The experiments also show that averaging all trajectories together can hide overconfidence at later stages, which becomes visible when results are analyzed across different horizons. In simpler terms, this paper identifies where the uncertainty comes from in the agentic system pipeline, how to teach agents to know when they are wrong, and why one confidence number is not enough.