Dual latent space control improves robot policy learning efficiency
Beyond Noise Steering: Dual-Latent Space Reinforcement Learning for Generative Robot Policy
Robotics
Summary
Robots that learn how to act by watching examples often use 'noisy' guesses that are hard to adjust precisely, which can slow down learning and reduce effectiveness. The authors propose a new method that lets the robot control both the starting guess and the details of the action as it develops. This double control helps the robot learn better and faster without changing its basic strategy. Their tests show this approach works well across different robots and tasks.
What this means in practice
- •For robotics engineers: Improve robot action learning speed and adaptability by using dual-latent control during policy generation in manipulation tasks.
- •For industrial automation teams: Enhance robotic manipulators’ performance in assembly or packaging by adjusting intermediate action features without retraining core policies.
Authors
Pengfei Zhang, Teng Sun, Xianchao Xiu
Abstract
Pretrained generative robot policies learn expressive action priors from demonstrations. However, existing reinforcement learning methods only steer the noisy space but fail to modulate intermediate action representations during the generation process, resulting in performance degradation and inefficiency. To address this limitation, we propose a novel Dual-Latent Space Reinforcement Learning (DLSRL) framework, which complements initial-noise steering with representation-level control inside the frozen generator. Specifically, our actor network predicts two distinct latent variables: an initial-noise latent variable that steers behavior generation, and an action-representation latent variable for intermediate feature modulation. Moreover, this representation latent variable is mapped to adapter features and ingeniously injected into the hidden states of intermediate action tokens via residual connections. Our dual-control design enables direct adjustment of action representations without updating the base policy. Experiments across generative policy architectures and robotic manipulation tasks show that DLSRL effectively accelerates online robot policy adaptation and achieves competitive performance. Our code is available at \href{https://github.com/xianchaoxiu/DLSRL}{https://github.com/xianchaoxiu/DLSRL}.