Robot programming system learns from single human demonstration
RAPID: Robot Agentic Programming from Demonstrations
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Programming robots to perform tasks usually requires lots of work and detailed instructions. The authors created RAPID, a system that watches one single video of a person doing a task and automatically writes, tests, and improves a robot's program for it. Instead of copying exact motions, RAPID focuses on the relationships between objects and their movements, so the robot can handle new situations with different object positions, shapes, or materials. The system was tested both in simulations and on a real robot arm and was able to perform complex tasks involving touching and moving objects without gripping them.
What this means in practice
- •For robotics engineers: Build robot programs from a single human demonstration that generalize across object variations without manual coding.
- •For manufacturing automation teams: Deploy robots able to perform complex object manipulation tasks with diverse materials and shapes guided by example videos.
Authors
Yuyao Liu, Jiayuan Mao, David Hsu, Leslie Pack Kaelbling, Tomás Lozano-Pérez
Abstract
Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.