TrajWAM improves dynamic robot manipulation by predicting object paths
TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation
Robotics
Summary
Robots often struggle to handle moving objects because they need to guess where the object will go and adjust their actions quickly. The authors created TrajWAM, a system that helps robots predict the future path of an object using past movement data, then plans actions based on this prediction. This approach speeds up decision-making by directly linking predicted movement with action planning, allowing robots to better keep up with fast-moving targets. Testing in both simulated and real settings showed TrajWAM works better and faster than previous methods.
What this means in practice
- •For robotics engineers: Enable robots to quickly plan and execute precise actions when manipulating moving objects in dynamic environments.
- •For industrial automation teams: Improve efficiency and accuracy of robotic arms handling fast-moving items on manufacturing lines using trajectory-guided control.$Commercial implications: Makes faster and more reliable robotic manipulation products possible for factories needing dynamic handling of moving parts.
Authors
Haohan Geng, Sha Zhang, Weishuo Sun, Guanting Ye, Wenhao Yu, Yuang Xie, Qicheng Zeng, Peng Ye, Jiajun Deng, Jianmin Ji
Abstract
Dynamic object manipulation requires robots to anticipate target motion while replanning fast enough to avoid executing outdated actions. Existing approaches either rely on fast reactive control without an explicit future target state or encode anticipated motion only implicitly. World Action Models (WAMs) offer a promising alternative by coupling future visual prediction with action generation, but scene-level video objectives may underrepresent target dynamics that are critical for manipulation, while iterative video-action denoising introduces substantial latency. We present TrajWAM, a trajectory-guided world action model for efficient dynamic robotic manipulation. Our key insight is to explicitly ground future visual imagination in predicted target-object motion and use the resulting future representation to condition action generation, while preserving this anticipation pathway after acceleration. Given a short history of object masks, a history-conditioned motion predictor forecasts future masks encoding the target object's trajectory. A lightweight Trajectory ControlNet then steers a pretrained video model toward trajectory-consistent futures. The resulting future representation conditions an action expert to predict executable action chunks. To enable one-step inference without breaking this interface, we introduce a decoupled distillation scheme that combines distribution matching for the video backbone, Trajectory ControlNet realignment, and action rollout distillation to support accurate one-step action prediction conditioned on the compressed future representation. Comprehensive evaluations in simulation and the real world show that TrajWAM achieves state-of-the-art performance on dynamic object manipulation tasks while enabling efficient one-step video-action inference. Project page: https://kevingeng3.github.io/TrajWAM/.