Summary
Training AI agents to use computers by interacting directly with software and tools is hard, risky, and slow in real life. The authors created a method called WildAWM that automatically builds computer environments combining visuals, commands, and services for training these agents more safely and flexibly. Their system checks to make sure tasks are solvable and challenging, producing many complex tasks that current AI models still find difficult. They trained AI models in these synthetic environments and saw improvements when the models tackled new, unseen computer tasks. This shows the approach helps AI learn better computer skills without needing real users or risky setups.
What this means in practice
- •For software developers: Create and test AI assistants that perform complex multi-step computer tasks across visual and command-line interfaces using synthesized training data.
- •For automation engineers: Develop more capable automation tools by training agents in diverse, controlled computer-use environments that simulate real-world interactions safely.
Authors
Zhaoyang Wang, Qianhui Wu, Haorui Wang, Simon Yu, Baolin Peng, Canwen Xu, Hao Cheng, Wenlin Yao, Boyi Liu, Yite Wang, Jianfeng Gao, Huaxiu Yao
Abstract
Agent learning is shifting from learning from human data toward learning through interaction with the world. However, scaling such learning directly in real-world computer systems is challenging and can introduce substantial security risks. We therefore propose Agent World Model in the Wild (WildAWM), a scalable pipeline for synthesizing controllable environments and tasks for training general computer-use agents (CUAs). Whereas existing approaches typically specialize in a single agent interaction interface, we construct executable environments that jointly support graphical user interfaces, model context protocol services, and command-line interfaces, allowing agents to flexibly combine visual interaction, tool use, and code execution. WildAWM is entirely powered by open-source models, demonstrating that high-quality computer-use environments can be generated without proprietary frontier models. Beyond environments, we address three recurring challenges in task synthesis: ensuring solvability, maintaining sufficient difficulty, and producing reliable verifiers, by introducing execution-based quality gates that combine heterogeneous agent rollouts, cross-judge consistency checks, and counterfactual probes of judgment robustness. The resulting pipeline synthesizes 341 long-horizon environment-task pairs that can require over 100 agent steps to solve and remain challenging even for frontier models, with GPT-6-Astra solving only 60% of them. We train two Qwen models of different sizes exclusively in these synthetic worlds. Both models achieve substantial gains across 8 out-of-distribution benchmarks spanning web, claw, and computer-use agents, demonstrating that learning in WildAWM's synthesized worlds generalizes broadly to unseen computer-use tasks. The code is publicly available at https://github.com/MSR-Orchard/WildAWM.