Computer-use agents perform many embodied ai tasks through software
Computer-Use Agents Are Embodied Generalists
Machine LearningRobotics
Summary
Embodied AI tasks usually need special robots or setups, but this paper shows that these tasks can be done by agents that interact with digital software like a human uses a computer. The authors built an agent that controls a cursor and keyboard to do embodied tasks by seeing screenshots, similar to how people operate computers. This agent won multiple competition prizes without any adjustment for specific tasks, showing that skills learned from software control apply broadly in embodied AI. The authors suggest this approach points toward general-purpose embodied AI agents.
What this means in practice
- •For robot developers: Build embodied AI robots using general agents trained on computer-based software interfaces to perform diverse tasks without task-specific tuning.
- •For software engineers: Create software-based AI agents that control virtual or remote environments through standard interfaces like screens and input devices.
Authors
Pranjal Aggarwal, Sean Welleck
Abstract
In many areas of AI, general-purpose models have increasingly replaced handcrafted methods and specialized architectures. We argue that embodied AI is poised for a similar shift. Our key claim is that embodied tasks can be formulated as interaction with digital software. Specifically, we build software that renders an agent's view of a physical environment on a screen and provides controls for acting in it. A computer-use agent (CUA) can then perform embodied tasks through the same screenshots and keyboard and mouse actions that it uses to operate ordinary software. Using a single CUA design and changing only the software, our entries win first-, second-, third-, and sixth-place prizes across four ECCV workshop competitions, without any competition-specific agent tuning. These results suggest that the skills CUAs learn on software, such as visual grounding, precise clicking, and exploration, already carry over to embodied tasks, giving CUAs a far broader scope in embodied AI than previously realized. We therefore advocate that computer-use agents can be embodied generalists. Further, we advocate that, by following the training strategy that made computer-use agents general, embodied AI can converge on a single general model as well, ushering in a new era of general embodied AI. Website: https://pranjal2041.github.io/cua-embodied