Action representation improves vision language robot performance
Direction-Scale Decomposition in Action Representation: Rethinking What to Tokenize for Vision-Language-Action Models
Computer Vision and Pattern RecognitionRobotics
Summary
Representing movements in robots is tricky because small changes can look very different depending on speed or how data is prepared. The paper introduces a new way to break down actions into their direction and size separately, making it easier for systems to understand and use motions across different tasks and datasets. This method helped robots perform better in both simulated environments and real-world tests, especially when learning from mixed sources of data. The authors show their approach reduces confusion and improves success rates in robotic manipulation.
What this means in practice
- •For robotic manipulation teams: Improve robot control accuracy by representing actions as separate direction and scale tokens to enhance success in diverse training settings.
- •For machine learning engineers: Develop vision-language-action models that better handle mixed datasets by using direction-scale decomposition for more robust action tokenization.
Authors
Yufei Duan, Hang Yin, Alberta Longhini, Chao Tang, Danica Kragic
Abstract
Action representation plays a central role in discrete-token vision-language-action (VLA) learning but remains underexamined. Under conventional pose-increment representations, action tokens are sensitive to execution speed and dataset-specific normalization, potentially obscuring geometric structure shared across demonstrations and datasets. We introduce Direction-Scale Decomposition (DSD), an action representation that decomposes translation and rotation increments into direction and scale components before tokenization. DSD isolates motion direction while retaining magnitudes in separate scale channels. We evaluate DSD with uniform binning (BIN) and BEAST, a B-spline-based tokenizer, in simulation and real-world manipulation under both single-dataset and mixed-dataset training. On LIBERO, DSD improves average success rates with both tokenizers. On SimplerEnv, DSD-BIN outperforms BIN by 10.3 percentage points in overall success rate under mixed-dataset training. Real-robot experiments further show gains both with and without robotics pretraining. These results support DSD as an effective action representation for discrete-token VLA models and suggest its potential to mitigate performance degradation when training on large and diverse dataset mixtures. Our project page with additional resources is available at https://vla-dsd.github.io/