Unified robot action space improves learning across different robot types
Cross-Embodiment Robot Foundation World Models with Latent Actions
Robotics
Summary
Robots come in many shapes and use different ways to move, which makes it hard to build a single computer model that works well for all of them. The authors created a new model that uses a shared 'hidden' way to represent actions for multiple robot types. This shared action space helps the model learn better and perform well even on robots it hasn't seen before. Tests showed their method does better than older ones, especially when handling more types of robots during training.
What this means in practice
- •For robotics engineers: Improve robot controllers by using models that perform better across different robot designs.
- •For robotics platform developers: Build unified training systems that adapt efficiently to new robot embodiments with varying motion styles.
Authors
Huang Huang, Sriram Yenamandra, Arjun Majumdar, Elie Aljalbout, Tushar Nagarajan, Tsung-Yen Yang, Akshara Rai, Michael Rabbat, Li Fei-Fei, Jiajun Wu, Tingfan Wu, Franziska Meier
Abstract
The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model's performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM's downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlight the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics. Project website: https://lacwm.github.io/