AI agents fail differently as their autonomy increases
When AI Agents Act on Our Behalf: A Taxonomy of Human-Agent Interaction Failure
Human-Computer InteractionArtificial IntelligenceComputers and Society
Summary
AI agents do tasks for people, like writing code or handling interfaces. This paper looks at how and when these agents fail depending on how much control they have. The authors found 12 main ways failures happen, and these change as agents act more independently. Because users' ability to fix problems depends on how much the agent does alone, the same AI can fail in different ways if set up differently. The paper shows users need options between doing everything themselves and fully trusting the AI.
What this means in practice
- •For software developers: Design AI tools with adjustable user controls to handle different failure modes at various autonomy levels.
- •For ui designers: Create user interfaces that support varying degrees of user involvement to prevent or mitigate AI agent failures.
Authors
Canwen Wang, Qing Xiao, Haiyi Zhu, Hong Shen
Abstract
AI agents write and execute code, manipulate user interfaces, and transact on their users' behalf. We ask what forms failure takes in human-agent interaction, and how those forms change as agents act with greater autonomy. From 695 documented incidents in two public repositories, we developed 12 failure modes bottom-up within three interaction stages adapted from Amershi et al.'s guidelines, coding the autonomy at which each arose. What a person can do when an agent fails depends on how much the agent completes between points of user involvement--a property fixed at deployment. Therefore, the same model on the same task can fail differently under different configurations. All 12 modes appear at both low and high autonomy, and rising autonomy changes the form each failure takes. Users need controls between full delegation and full manual work, and neither incident records nor agent benchmarks reach the interaction where these failures occur.