Evaluating how AI agents admit uncertainty when facing conflicting knowledge

Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict

Artificial IntelligenceComputation and Language

Summary

This paper looks at how AI agents handle situations when what they 'know' conflicts with new evidence they find. The authors check if agents change their answers, admit they’re unsure, or stubbornly stick to wrong conclusions. They find that even when agents get answers right more often, they don’t always admit uncertainty or explain their doubts. Also, agents might notice conflicts early on but fail to resolve or communicate them properly later. Adjustments to the underlying models can help agents be more humble but sometimes make them less accurate.

What this means in practice

  • For ai system developers: Improve conversational AI by designing agents that better acknowledge uncertainty when evidence conflicts with their knowledge.
  • For software reliability teams: Detect when AI systems inconsistently address uncertainty during multi-step tasks to enhance system trustworthiness.
  • For customer support automation teams: Develop chatbots that clearly communicate uncertainty when provided information conflicts with pre-trained knowledge, improving user trust.$Commercial implications: Supports building customer-facing AI that transparently manages conflicting information, a key commercial differentiator in support solutions.

Authors

Kaiser Sun, Bernal Jimenez Gutierrez, Hongjun Liu, Jingyu Zhang, Jie Gao, Mark Dredze, Daniel Khashabi

Abstract

When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusion? Existing evaluations of agentic systems focus primarily on task success, offering limited insight into how agents handle such conflicts. We propose to evaluate agents on epistemic humility (EH): the agent's willingness to recognize, act on, and communicate uncertainty during task execution. We operationalize EH through three trajectory-level behavioral dimensions: Identify, Solve, and Escalate (ISE). Through knowledge conflict, situations where the backbone language model's parametric knowledge contradicts the evidence it encounters, or where two contextual sources disagree, we evaluate two conflict settings: (1) controlled conflict and (2) naturally occurring conflict during multi-step agentic execution, each paired with matched no-conflict controls. Evaluating four agents, we find that higher task accuracy does not necessarily correspond to greater epistemic humility: some high-accuracy configurations recognize conflicts during execution but do not communicate unresolved uncertainty in their incorrect final answers. Trajectory-level analysis further reveals that agents frequently detect conflicts in early steps of execution but fail to maintain or resolve them in later steps. Finally, we show that model-level interventions can improve EH, but often at the cost of task accuracy, suggesting that epistemic humility emerges from the interaction among the backbone model, agent harness, and evaluation environment.