Computer use agents improve tasks by combining gui and command line
HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents
Computer Vision and Pattern Recognition
Summary
Computer use agents help people complete tasks on computers automatically, but they usually use either only the graphical interface or special apps that need lots of programming. This paper shows a better way by teaching agents to use both the graphical interface and command line together, taking advantage of each method's strengths. The authors created a large dataset to teach these agents and trained them to know when to use commands to be more efficient. Their improved agent works better across different computer systems, proving this mixed approach is useful.
What this means in practice
- •For software automation engineers: Develop automation tools that switch between GUI and command line to complete tasks more efficiently across different operating systems.
- •For it operations teams: Improve system management by enabling agents that choose the best interaction method for tasks to reduce errors and speed up workflows.
Authors
Tongbo Chen, Junbo Niu, Zhengxi Lu, Niu Lian, Fei Tang, Yuchen Yan, Yike Hong, Yong Du, Yizhou Liu, Bofan Chen, Yongliang Shen
Abstract
Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often inefficient and error prone, or augment GUI interactions with application specific APIs or tools, which require substantial engineering effort and are difficult to scale across applications. We argue that the next generation of CUAs should combine GUI interactions with the command line interface (CLI), leveraging the generality of the GUI and the efficiency of shell commands. A critical challenge, however, is that current models do not know when or how to use the CLI during task execution. To address this challenge, we develop a data construction pipeline that produces three types of trajectories: GUI only, CLI only, and interleaved GUI and CLI trajectories. This pipeline results in HybridCUA-8K, containing 5K hybrid trajectories and 3K verified RLVR tasks. Building on these data, we propose a training framework with two stages: supervised fine tuning on the constructed trajectories, followed by reinforcement learning with our CLI aware rewards that encourages agents to use the CLI selectively and reliably. Experiments show that HybridCUA-9B achieves 53.6% accuracy on OSWorld, improving over the base model by 14.8 percentage points, and improves performance on WindowsAgentArena by 4.0 percentage points. These results demonstrate the effectiveness and cross platform generalizability of the hybrid GUI and CLI paradigm for computer use agents.