Coding agents improve robot learning for long tasks and dexterity
EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics
Robotics
Summary
Some robots need to perform long and complicated tasks involving careful handling of objects. The authors created a computer program benchmark to test coding agents that can generate step-by-step robot actions for these tasks. They found these agents can solve hard tasks and reuse solutions but require many tries and often focus only on one task. To make this easier to use for teaching robots, they expanded one solution into many examples that a learning robot can practice with. Robots trained this way did better on new tasks and even succeeded in real-world tests. This approach helps combine programming and learning for robot skills.
What this means in practice
- •For robotics engineers: Create complex robot policies that handle long tasks with dexterity by generating diverse training data from coding agent solutions.
- •For automation developers: Train robots on realistic multi-step manipulation tasks in simulation that transfer successfully to real hardware.
Authors
Haoxiang You, Zeyu Shen, Yilang Liu, Zhicheng Zheng, Lihan Zha, Kashu Yamazaki, Mingtong Zhang, Suning Huang, Jiankai Sun, Qianzhong Chen, Lucy He, Kaiyuan Liu, Haoran Chang, Katerina Fragkiadaki, Dhruv Shah, Mac Schwager, Peter Henderson, Ian Abraham, Canwen Xu
Abstract
We study coding agents for long-horizon, dexterous robotics and ask whether their solutions can provide scalable supervision for learning general robot policies. To test this, we develop EMBODIEDSWE-BENCH, a simulation benchmark for coding agents spanning contact-rich manipulation, deformable objects, and long-horizon tasks requiring up to half an hour of continuous interaction. We find that frontier coding agents can solve complex long-horizon tasks and transfer prior solutions across both tasks and embodiments. We also design supporting tools that help agents more effectively solve these tasks. However, the resulting solutions require substantial iterative interaction and are typically specialized to individual task instances. We therefore introduce EMBODIEDSWE-GEN, which expands a single solution from coding agent into large diverse trajectories for training a VLA. VLA performance improves with more generated demonstrations, and agent-aided diversification improves generalization to held-out task variations. We also show that a VLA finetuned solely on coding-agent-generated simulation demonstrations completes a long-horizon task on real robot. Together, our framework uses coding agents to solve complex robotics tasks and turn verified solutions into scalable supervision for robot policies.