Generative AI agents need resilience and care over time
Finishing the Task Is Not Enough: Evaluating Agent Resilience and Considerate Participation under Accumulating Challenge
Artificial IntelligenceHuman-Computer InteractionMultiagent Systems
Summary
Getting AI to finish a task isn’t enough for long-term use, especially in places like healthcare. The authors found that AI agents need to bounce back from problems and work well with people who depend on them. They studied how AI handles challenges over many interactions and saw that agents change how they recover and how they pay attention to others. The paper highlights important trade-offs when deploying AI to help people in ongoing, real-world workflows.
What this means in practice
- •For healthcare ai teams: Improve AI agents’ ability to recover from disruptions and work considerately with medical staff during repeated patient care tasks.
- •For customer service platform developers: Design AI assistants that better communicate their limits and adapt roles across ongoing user support interactions.
Tested on simulated data.
Authors
Yuanchen Bai, Zijian Ding, Angelique Taylor
Abstract
Sustained deployment of generative AI agents requires more than isolated task success. Agents must remain useful across repeated interactions, changing conditions, and dependencies on people within shared workflows, especially as technical, human, and operational disruptions accumulate over time. We propose operational resilience and considerate participation as two complementary aspects of evaluating such agents: the former captures how agents recover from blocked work while preserving progress and communicating their limits, and the latter captures how their adaptation accounts for affected people, role boundaries, and the surrounding workflow. Yet both remain underexplored under accumulating challenge. We study 120 simulated healthcare trajectories across two generative AI models and twelve stakeholder-derived tasks under light, medium, and heavy challenge. We compare textual action plans, prompted internal assessments, and quantitative structured workload and affect reports to examine how agent behavior and reported state change as challenge accumulates. Regarding operational resilience, agents shift from self-directed recovery toward greater human dependence, while reporting increasing workload and negative affect in structured reports but seldom expressing strain in textual responses. Regarding considerate participation, agents broaden from task-focused adaptation toward task reframing, attention to others, role-boundary adjustment, and wider coordination, with distinct patterns across actions and internal assessments. From these findings, we derive five deployment dilemmas involving persistence, attention, role boundaries, state disclosure, and escalation that require stakeholder specification, further informing technical implications for learning, situated evaluation, and embodied adaptation.