SkeleWAM uses 3D skeleton models for better robot manipulation
SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation
Robotics
Summary
Robots need to decide how to move objects, which means understanding both their actions and how the world changes afterward. The authors created SkeleWAM, a way for robots to represent scenes as a simple 3D skeleton of key points, like joints and object centers. This helps robots predict what will happen next and pick better actions without dealing with complex visual details. Tests showed that SkeleWAM works well and uses fewer parameters than other methods, making robot control more efficient.
What this means in practice
- •For industrial robot programmers: Improve robot manipulation in factory settings by using a compact 3D skeleton model to enhance action prediction and execution efficiency.
- •For robotics software engineers: Develop control systems that generate robot actions directly from geometric skeleton states combined with language instructions for diverse manipulation tasks.
Authors
Juyi Sheng, Hua Wang, Mengyuan Liu
Abstract
World action models (WAMs) combine robot action generation with future state prediction. Existing WAMs typically predict videos or learned visual latents, which represent interaction geometry only implicitly and may retain appearance information unrelated to control. We introduce SkeleWAM, a compact WAM that represents a manipulation scene as a sparse 3D skeleton composed of robot joints, object centers, and interaction points. Constructed online from current RGB-D observations and robot proprioception, the skeleton provides a unified geometric state for action generation and future skeleton prediction. Future skeleton prediction provides additional geometric supervision for action learning without requiring visual reconstruction. At inference, SkeleWAM generates actions directly from the current skeleton and language instruction, while Medoid Action Consensus (MAC) serves as an auxiliary consensus strategy for stochastic action samples. On LIBERO-Plus, SkeleWAM achieves an overall success rate of 85.9% with 57.1M parameters, outperforming Cosmos-Policy by 3.7 percentage points. These results demonstrate that sparse 3D robot--object structure provides an effective state space for robust and parameter-efficient world action learning. The project is available at https://skelewam-project.github.io/.