PointWAM improves 3D robot hand actions using human videos
PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
RoboticsComputer Vision and Pattern Recognition
Summary
Robots need to understand how things in the world move and how their own hands should move to interact with objects well. The authors created a new way called PointWAM that shows both the scene and robot hands as 3D points moving together over time. By learning from real human video demonstrations, their method helps robots predict better hand movements for complex tasks. This approach leads to much higher success in robot hand tasks compared to older methods.
What this means in practice
- •For robotics engineers: Use PointWAM to improve robot hand manipulation accuracy by jointly predicting scene and hand movements in 3D.
- •For robotic system integrators: Integrate PointWAM pre-trained on human videos to enhance dexterous task success rates on real robotic platforms.
Authors
Chunghyun Park, Beomjun Kim, Seungcheol Park, Heeseung Kwon, Yashu Shukla, Seunghoon Sim, Jinwoo Shin, Minsu Cho
Abstract
World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.