Users often miss hidden AI errors despite high confidence
Looks Fine to Me: Understanding Oversight in Agentic Interactions
Human-Computer Interaction
Summary
When people use AI systems to help with tasks like programming, they often trust the AI outputs without checking carefully. The authors found that users usually do simple checks and feel confident even when the AI makes hidden mistakes that aren’t obvious. These mistakes can be overlooked because the output looks plausible and people don’t always know when to look closer. The study suggests that better ways are needed to help people know when to pay more attention to AI results.
What this means in practice
- •For software developers: Improve user interfaces by signaling when AI-generated code might contain subtle errors requiring closer inspection.
- •For data analysts: Design better oversight tools to catch plausible but incorrect AI analysis outputs during data-driven decision processes.
Authors
Dinithi Dissanayake, Yasith Samaradivakara, Hyung Woon Lee, Yadeesha Weerasinghe, Sanjay Saha, Pattie Maes, Suranga Nanayakkara
Abstract
Agentic AI systems are deployed to execute complex tasks across domains such as writing, coding, and data analysis. As execution is offloaded to AI agents, the human role shifts toward oversight. Effective oversight requires recognizing when closer scrutiny is needed, even though flawed outputs may appear convincing. Our formative study (N=12) revealed that users accepted plausible outputs without thorough evaluation, and only recognized these lapses in retrospect. To examine this systematically, we conducted a controlled within-subject study (N=36) in which participants completed programming tasks with an agent that produced correct outputs, silent defects, or visible failures. Combining task outcomes, interaction logs, gaze, physiological signals, and self-reports, we examined perceived, behavioral, and effective oversight. Participants were often highly confident while missing silent defects: they performed generic checks, failed to generate diagnostic evidence, or even misinterpreted defect evidence. We discuss how studying oversight across multiple dimensions can support future work on timely, task-calibrated re-engagement.