Computer-use agents improve reliability with goal-oriented error recovery
Lost in Recovery: Demystifying Recovery Behavior of Computer-Use Agents via Goal-Oriented Action Replay
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Digital assistants that operate software by imitating mouse and keyboard actions often make mistakes that add up over time, causing problems. The authors introduce a way to study how these agents can recover from errors without restarting the system, by replaying actions aimed at reaching a goal despite earlier mistakes. Their work shows that when these agents recover well, they complete tasks better, but current agents do not use this ability effectively. Accurate recovery methods lead to noticeable improvements in how well these agents perform from start to finish.
What this means in practice
- •For software automation engineers: Improve automated workflows by integrating recovery methods that correct errors without restarting the environment, enhancing agent reliability for complex tasks.
- •For desktop application developers: Design applications with support for goal-consistent action replay to enable automated agents to recover from unintended user actions safely.
- •For robotic process automation providers: Develop more robust RPA tools that leverage model-based recovery to reduce failures and improve task completions in proprietary software environments.$Commercial implications: Enables creation of commercial automation products that handle errors autonomously across complex software workflows.
Authors
Byeonghun Bang, Sangwoo Park, Jinheon Baek, Kangsan Kim, Youngwan Lee, Sung Ju Hwang
Abstract
Recent advances in large (vision) language models have enabled computer-use agents that automate digital workflows via fine-grained keyboard and mouse operations. However, such agents remain fragile in long-horizon settings, where errors accumulate and executed actions leave unintended side effects, making the ability to recover from failures critical. Yet, existing remedies remain inadequate: environment resets are infeasible in proprietary settings, action replay is prohibitively expensive over long horizons, and API-based rollback is limited to web environments, leaving model-based recovery largely unexplored. To address this gap, we present ReTrace, a scalable evaluation framework for systematically studying recovery behavior in computer-use agents. We formalize Transition-Consistent Recovery: identical downstream actions from the recovered and target states must yield identical outcomes. We operationalize this criterion through goal-oriented action replay following controlled error injection. Using ReTrace, we present the first systematic study of autonomous, model-based recovery, revealing that (i) ideal recovery can unlock model's latent task completion capabilities, (ii) current computer-use agents fail to exploit their latent recovery capabilities, and (iii) accurate model-based recovery yields measurable end-to-end gains. Together, these findings establish recovery as a fundamental capability for building reliable computer-use agents. Our code and dataset is available at https://github.com/BangProx/ReTRACE.