Local Motion Matters: A Deconstruct-Recompose Paradigm for Reinforcement Learning Pre-training from Videos
2026-07-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors found that instead of looking at an entire robot's movement as one whole, it's better to break it down into small local actions. They created a method called the Deconstruct-Recompose Paradigm that first identifies these small movements and learns from them separately. Then, they combine these learned parts to understand overall motion better, helping robots learn tasks faster and more efficiently. Their experiments showed this way works well for different robots and tasks.
reinforcement learningpre-traininglocal motion representationspatiotemporal relationshipslatent dynamics modelDual-Attention EncoderMotion Aggregation Tokenrobotic controlpolicy learningsample efficiency
Authors
Jinwen Wang, Youfang Lin, Xiaobo Hu, Shuo Wang, Kai Lv
Abstract
Pre-training on large-scale videos to improve reinforcement learning efficiency is promising yet remains challenging. Existing methods typically treat the agent as an indivisible entity, modeling motion patterns globally. Such global modeling is tightly coupled with the morphology, hindering transfer across domains. In contrast, despite the vast disparity in global motions, the local components exhibit similar motion patterns across different agents. Building on this insight, we propose a novel Deconstruct-Recompose Paradigm (DRP) for learning transferable local motion representations. Specifically, in the Deconstruct phase, we identify multiple local points and track their frame-wise motions, defining each as an Atomic Action. We introduce a Dual-Attention Encoder (DAE) to learn local motion representations from these Atomic Actions, capturing their spatiotemporal relationships. In the Recompose phase, we compose local motion representations with a learnable Motion Aggregation Token [MAT] via latent dynamics model learning. Additionally, an adapter bridges local motion and downstream action-specific dynamics to accelerate policy learning. Extensive experiments demonstrate that our method effectively transfers to diverse robotic control and manipulation tasks, significantly improving sample efficiency and performance.