Action representations affect control success beyond reconstruction error
Beyond Reconstruction Error: Analytical and Data-Driven Action Tokenization for Autoregressive Vision-Language-Action Models
RoboticsMachine Learning
Summary
When computers learn to perform tasks by imitating actions, they often measure how well they can recreate those actions. This study shows that simply measuring how accurately an action is copied does not guarantee good control in real tasks. The authors tested different ways to represent actions and found that things like predictability and stability of action sequences matter just as much. This means better ways to evaluate action representations are needed to improve how computers plan and act.
What this means in practice
- •For robotics engineers: Select and evaluate action representations to improve robot task success rates beyond just measuring reconstruction accuracy.
- •For machine learning developers: Design autoregressive models for vision-language-action tasks that consider sequence predictability and decoder stability for better performance.
Authors
Yuxin Yang, Gaohan He, Changxue Guan, Hangming Liu
Abstract
Discrete action tokenization is central to autoregressive vision-language-action (VLA) models, yet action representations are often evaluated primarily through reconstruction fidelity. We ask which representation properties actually matter for closed-loop control by comparing fixed analytical, data-driven linear, and nonlinear neural representations under a unified tokenization interface. Across rate-distortion analysis, sequence-modeling diagnostics, and 3,500 LIBERO rollouts, representation rankings change with the evaluation criterion. PCA achieves lower nominal reconstruction error than Temporal-DCT, but produces less predictable token sequences and 3.0 percentage points lower mean seen-task success across three policy-training seeds, with the policy ordering reversing in one seed. In a matched seed-42 ablation, an autoencoder further reduces reconstruction error yet does not yield the strongest policy and exhibits greater sensitivity to discrete token perturbations. These findings show that reconstruction fidelity alone cannot reliably select action representations for autoregressive control, motivating joint evaluation of geometric fidelity, sequence predictability, decoder stability, and closed-loop performance.