Trace2world reconstructs app behavior for scalable gui agent training
Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training
Machine LearningSoftware Engineering
Summary
Training AI to interact with apps can be tricky if the training environments don't behave like the real apps. The authors of this paper created Trace2World, a system that builds realistic, runnable app environments from real user interaction traces. This helps AI learn better by practicing in more realistic simulations that reflect actual app behavior. Their approach improves the accuracy of reproducing tasks and boosts AI training performance across multiple benchmarks.
What this means in practice
- •For mobile app developers: Create accurate simulated environments from real user traces to train and test interactive agents more reliably.
- •For software testing teams: Generate executable app mocks from interaction traces to automate and validate complex workflows with closer-to-real behavior.
- •For robotic process automation engineers: Build trustworthy app task models from observed interactions to train automation agents that handle GUI workflows effectively.$Commercial implications: Enables sale of robust automation tools that reduce manual setup by providing high-fidelity modeled environments for training.
Authors
Zezhou Wang, Hanzhang Zhou, Panrong Tong, Quyu Kong, Pengxiang Li, Steven Hoi, Zhuzhong Qian, Yue Wang
Abstract
Scaling GUI reinforcement learning requires environments that preserve real-app behavior. Yet coupling behavior inference with coding can turn partial traces into runnable mocks with incorrect state transitions. We introduce \textbf{Trace2World}, which separates global behavior reconstruction from implementation. It reconciles traces into a shared, source-linked app model, then compiles an executable environment. Independent replay of source workflows localizes mismatches for bounded implementation repair. Validated worlds then support learner-adaptive task and environment augmentation within source-supported behavior. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1% are reproduced end to end, versus 59.8% for the strongest matched task-local baseline. Source-grounded validation retains 94.0% of simulated training gains on real devices, versus 54.7% under self-validation. Learner guidance raises benchmark success from 56.9% to 61.1%. Training in Trace2World further improves the model by 20.2 points on our benchmark, 5.4--16.2 points across four external GUI benchmarks, and 16.3 points on 100 real-device tasks. Reliable reconstruction separates what behavior source evidence supports from whether the implementation preserves it.