Computer use agents improve new software tasks with user manuals
Adaptation of Computer Use Agents to New Software with User Manuals
Artificial Intelligence
Summary
It can be hard for computer programs that control software by watching and clicking to work well with new or updated applications. The authors propose a new method called MAGE that uses user manuals to guide these computer agents on what to do, helping them understand complex tasks better. MAGE also learns from interacting with the software itself to handle differences between manuals and actual screens. This approach reduces the need for expensive human-created training data and works well across several different software applications.
What this means in practice
- •For software developers: Enhance automated testing tools by enabling agents to understand new software workflows from manuals.
- •For enterprise automation teams: Create adaptive automation agents that can quickly learn to use new business applications with minimal manual input.$Commercial implications: Enables commercial deployment of adaptable automation bots for diverse business software, reducing manual setup costs.
Authors
Chulhoon Yoon, Kyungri Park, Jinyeong Yun, Daeyun Jeong, Sukmin Yun, Woohwan Jung
Abstract
Recent advances in multimodal foundation models have significantly improved the ability to operate real-world software through graphical user interfaces (GUIs) using visual observations and natural language instructions. However, maintaining strong performance on unseen applications remains challenging, as practical tasks often require long-horizon procedural knowledge and application-specific operation patterns. In addition, constructing executable tasks and collecting corresponding trajectories for training still heavily rely on human-annotated data, incurring substantial cost. In this paper, we propose MAGE (Manual-augmented computer use AGent Execution and learning), a framework that leverages user manuals as a scalable source of procedural knowledge to improve both inference and training. During execution, MAGE retrieves task-relevant manual content and uses a planner to generate explicit procedural guidance for CUAs. To further account for the discrepancies between manuals and actual GUI environments, such as version differences or user settings, MAGE also adapts the agent policy through environment interaction. During training, MAGE converts manuals into executable task-plan pairs and uses the plans to guide environment rollouts toward task-relevant states; the resulting trajectories are then used to optimize the policy without human-authored demonstrations. This unified design enables effective utilization of manuals for both task execution and training, reducing reliance on human-authored training data. Experimental results across six applications demonstrate improvements in task success rates over strong baselines, highlighting the effectiveness of manual-guided planning and policy adaptation in new software environments. We provide the code and reproduction scripts at https://github.com/ku-autoai/MAGE.