Principal steering subspaces speed up robot policy adaptation online
Principal Steering Subspaces for Online Adaptation of Frozen Generative Robot Policies
Robotics
Summary
Robots can have complicated ways to decide what to do, but changing these ways while the robot is working can be slow and hard. The authors found a simpler way to control the robot by focusing on just the most important directions of change, which makes learning faster and more stable. They tested this on several robot tasks and even on a big vision-language robot system, seeing better results after using their method. This suggests that understanding how robot decisions respond to small tweaks helps adapt robots more efficiently.
What this means in practice
- •For robotics development teams: Improve online learning speed and stability for robots using large frozen generative models by controlling key latent directions.
- •For industrial automation engineers: Enhance robot manipulation tasks like screwdriver placement by adapting complex policies without retraining entire models.
Authors
Jialeng Ni, Nathan Zhao, Kunpeng Song
Abstract
Generative robot policies provide expressive behavior priors, but updating a large diffusion or flow-matching model through online interaction is costly. Latent-space reinforcement learning avoids updating the pretrained generator by controlling its initial sampling noise, yet high-dimensional noise can have strongly anisotropic effects on decoded actions. We introduce Principal Steering Subspaces (PSS), a forward-query interface that constructs a fixed low-dimensional control basis from finite-difference decoder responses. Soft Actor-Critic controls the leading response directions, while the orthogonal complement is independently resampled from the Gaussian prior at each query. On three RoboMimic tasks with diffusion and flow-matching policies, response spectra reveal substantial concentration. Across five matched task-generator pairs, the training curves indicate that PSS generally converges faster and exhibits more stable late-training behavior than full-latent control, while achieving stronger final performance overall. Controlled Diffusion-Square ablations further show that leading-response directions outperform random and least-responsive subspaces of equal dimension. We further integrate PSS with a frozen, closed-source 3B-parameter vision-language-action (VLA) policy in a humanoid learning system with synchronous transition collection, reset-time optimization, and latency-aware asynchronous deployment. In an exploratory screwdriver-placement evaluation, success is observed in 2/10 trials for the frozen VLA policy and 6/10 after SAC+PSS adaptation. These results support decoder-response geometry as a practical basis for online adaptation of frozen generative robot policies.