JarvisGUI enables coordinated workflows across multiple devices
JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
Artificial Intelligence
Summary
Many tasks people do on computers involve working across different devices like phones, laptops, and desktops, but most testing for computer helpers only looks at one device at a time. The authors created JarvisGUI, a system that tests how well these helpers can work across several devices and operating systems by combining tasks in new ways. They found that current helpers struggle with remembering what happened on one device when moving to another and managing complicated, multi-step tasks. This shows that these helpers need to improve before they can help with real-world multi-device workflows.
What this means in practice
- •For software testers: Evaluate tools that automate GUIs across different devices to uncover weaknesses in handling multi-step, cross-platform workflows.
- •For enterprise it teams: Improve internal automation by assessing and enhancing agents that transfer and coordinate tasks across various company devices.
Authors
Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, Yuhang Guo, Yunhong Wang, Haifeng Wang
Abstract
Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device workflows and dynamically evaluate agent performance within a unified framework. By evaluating agents in virtual environments spanning multiple operating systems, JarvisGUI reveals that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world workflows, exposing a critical capability gap invisible to existing benchmarks.