Trust can limit effective human AI decision making under uncertainty
Trust Hinders Bidirectional Human-AI Alignment
Artificial IntelligenceHuman-Computer Interaction
Summary
People usually try to trust AI systems just the right amount when making decisions, but this idea doesn’t work well in complicated situations where not everything is known or clear. The authors show that relying solely on trust is too narrow and that people use other ways to handle uncertainty besides trust. They suggest that alignment between humans and AI needs to consider more complex decision processes to work better, especially in unclear or open-ended tasks.
What this means in practice
- •For product managers: Design AI-assisted decision workflows that go beyond trust calibration to better handle uncertain or unclear user goals.
- •For customer support teams: Implement AI tools that support agents in navigating ambiguous situations by jointly exploring uncertainties rather than just trusting AI outputs.
A position paper. It proposes an approach and reports no results.
Authors
Michaela Benk, Kacper Sokol
Abstract
Trust calibration remains the dominant framing of AI-assisted decision making: the goal is to close the gap between perceived and actual trustworthiness of AI systems so that users exhibit appropriate reliance on them. We argue that this approach, while useful for decision problems that are well-defined and clearly scoped, overly restricts what it means to make a decision with AI support. To overcome this limitation, we develop three dimensions of AI-assisted decision making. First, trusting AI is only one of several possible strategies for managing uncertainty. Second, trust calibration is often misapplied when AI models "large-world" decision problems, i.e., ill-defined situations where goals and states are only partially known or unknowable. Third, in such settings human-AI alignment requires addressing higher-order decision processes. We conclude by reflecting on the implications of our observations on the design of process-oriented human-AI coupled systems that jointly identify and negotiate uncertainty throughout the decision-making task.