General purpose agent directly controls robots for varied tasks
Agent as Policy for Robotic Manipulation
Computation and Language
Summary
Robots often need special training for each new task or environment, but this paper shows that a general-purpose agent can control a robot for many different tasks without that special setup. The agent looks at what the robot sees, writes instructions, and adjusts as things happen in the real world. The researchers tested this system on things like building with blocks, folding towels, and throwing objects, achieving high success rates. This means one smart agent can handle a variety of robot tasks by thinking and programming on the fly.
What this means in practice
- •For robotics engineers: Develop robots that perform complex manipulation tasks without task-specific training by using a general agent that plans and adapts in real time.
- •For warehouse automation teams: Automate varied item handling and assembly tasks with fewer custom programs by deploying a single agent that interprets goals and controls robot motions dynamically.
Authors
Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, Meng Jiang
Abstract
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. AGP achieves success rates of 100%, 100%, and 80% on three block construction configurations. These findings establish a path for general-purpose agents to act as robotic policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.