Action centric learning improves visual robot planning accuracy
Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling
Computer Vision and Pattern Recognition
Summary
Some computer programs learn how actions cause changes just by looking at videos, but they often focus too much on copying images rather than understanding true actions. The authors found that better image copying does not always mean better understanding of actions. They created a new method called ACT-LAM that better separates action signals from unrelated video details and uses those actions more effectively to predict what happens next. This method works well on robot video data and helps robots plan their moves more accurately.
What this means in practice
- •For robotics engineers: Improve robot control systems by training models that better understand and predict the effects of robotic actions from raw video.
- •For autonomous vehicle developers: Enhance vehicle planning algorithms by more accurately modeling the impact of driving actions from visual inputs.
Authors
Dingjie Fu, Dianxing Shi, Yangyang Xu, Jun Yu
Abstract
Latent action models (LAMs) learn action representations from unlabeled videos by inferring latent actions from visual transitions and reconstructing future states. However, we identify a fundamental $\textbf{reconstruction-action mismatch}$: lower reconstruction error does not necessarily yield better latent dynamics or downstream performance. We attribute this mismatch to two underconstrained aspects of reconstruction-based latent dynamics modeling: (i) the inverse dynamics model (IDM) is not explicitly encouraged to distinguish action-related transitions from nuisance appearance, and (ii) the forward dynamics model (FDM) can underutilize the inferred latent action by exploiting predictive shortcuts from the current state. To address both limitations, we propose $\textbf{ACT-LAM}$, a lightweight action-centric framework that strengthens both action extraction and action utilization. Specifically, its Action Query IDM (AQ-IDM) employs learnable action queries and gated aggregation to selectively extract rich action-related transition cues without strong information bottlenecks. And its Action Token FDM (AT-FDM) projects latent actions into action tokens that progressively interact with evolving state representations, enabling continuous state-aware action conditioning. ACT-LAM further streamlines feature processing to concentrate model capacity on latent dynamics modeling. Extensive experiments on several robotic datasets and the VP$^2$ benchmark demonstrate stronger latent action consistency, forward dynamics, and downstream visual planning performance with fewer trainable parameters and lower computational overhead. In particular, ACT-LAM surpasses the previous state of the art by $\textbf{7.6%}$ on the aggregated VP$^2$ success rate. Codes at $\href{https://github.com/DingjieFu/ACT-LAM}{url}$.