Robot model uses effect programs to generalize object actions across arms
Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models
Robotics
Summary
Robots learn how objects move from human demonstrations by focusing on what happened to the objects, not the robot arm that caused it. The authors present PEWAM, a large model that breaks down demonstrations into programs representing object movements, grips, and final scenes without showing the demonstrator. This lets PEWAM generate robot actions tailored to different robot arms and respond to changes in real time, even improving success on some tasks compared to previous methods. People can also edit these object movement programs to adjust robot actions, making the approach flexible and reusable.
What this means in practice
- •For industrial robot programmers: Run single effect programs across different robot arms without retraining, enabling flexible reuse of learned object manipulation tasks in factory settings.
- •For robotics software developers: Edit and re-solve effect-based programs to adapt object manipulation actions in real time, improving robustness after unexpected scene changes.
Authors
Junyi Hu, Zhewen He, Zhenhua Li, Yi Fang
Abstract
A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.