AI explanations can increase novice over-reliance without feedback
Novice Reliance Calibration in AI-Assisted Decision Making: The Role of Explanations and Self-Assessment
Human-Computer InteractionArtificial Intelligence
Summary
When people use AI tools to help make decisions but don’t get immediate feedback, they may start relying too much on the AI, especially if the AI provides explanations. The researchers studied how beginners adjust their trust in AI during a clinical task and found that explanations can actually cause users to overtrust the AI. However, people who felt they understood the task better were more careful about when to trust the AI. This study helps understand how to design AI tools that avoid users blindly trusting AI in situations without clear feedback.
What this means in practice
- •For clinical software developers: Design AI-assisted clinical tools that manage user trust when immediate correctness feedback is unavailable.
- •For enterprise ai product teams: Create AI explanations that reduce over-reliance risks in decision support tools without real-time user feedback.
Authors
Eun Jeong Kang, Peter, Duan, Swati Mishra
Abstract
Artificial Intelligence (AI) tools are widely used to support decision making in tasks and domains where no immediate performance feedback is available. In these settings, users cannot learn to adjust their reliance behavior over time through trial and error. However, little is known about how novice users calibrate reliance on AI when external feedback is unavailable, or whether AI explanations can support calibration in its absence. We introduce reliance calibration as an organizing construct for studying how novice users dynamically adjust reliance behavior, and examine how AI explanations and meta-cognitive self-assessment shape it. Through a between-subjects study with 110 participants completing a clinical entity extraction task with AI assistance and limited performance feedback, we observe that novice users exhibit systematic drift toward over-reliance in the presence of explanations, while higher self-reported task understanding is associated with more selective reliance behavior. These results extend reliance calibration research into human-AI collaboration contexts without real-time performance signals and present actionable guidelines on designing AI tools that must support appropriate reliance in these settings.