AnchorQ improves robot policy learning with fewer trials
Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors
RoboticsArtificial Intelligence
Summary
Teaching robots to perform tasks better can take many tries, which is costly and slow. This paper introduces AnchorQ, a new way to help robots learn by focusing exploration more effectively using a structured approach called anchors. The method helps the robot adjust its actions step-by-step, making learning more efficient. Tests in both computer simulations and real tasks show AnchorQ helps robots succeed much more often using the same amount of practice time.
What this means in practice
- •For industrial robotics teams: Improve robotic arm performance in manufacturing lines with less trial-and-error adjustment.
- •For automation system integrators: Refine deployment-level robot behaviors efficiently for new tasks or environments after initial training.
- •For robotics hardware manufacturers: Embed learning algorithms that increase task success rates under limited interaction budgets in robot products.$Commercial implications: This paper enables selling robots with advanced learning capabilities that improve themselves efficiently in real settings.
Authors
Carl Qi, Weijun Dong, Xiaojie Wang, Peiyi Chen, Rahul Sajnani, Mingyuan Zhang, Amy Zhang, Jose Barreiros, Manikatan Nambi, Lillian J. Ratliff, Dylan Randle
Abstract
Generalist robot policies offer strong behavioral priors, but achieving deployment-level reliability and throughput remains difficult. Reinforcement learning (RL) can improve these policies, yet its effectiveness is constrained by the cost of online interaction. Existing methods adapt frozen flow-matching policies by treating noise space as a continuous action space. We show that this behaviorally unstructured space makes exploration inefficient, often preventing these methods from extracting meaningful learning signals on challenging tasks. We introduce AnchorQ, which enables coarse-to-fine policy improvement through a structured, refinable latent action representation. At each decision step, AnchorQ uses flow reversal to map reference behaviors to state-conditioned latent anchors, samples local particles around those anchors, and applies value-guided sampling to explore promising refinements. Our analysis shows that anchors reduce identification complexity at the cost of support bias; particle refinement trades this bias against action-class complexity; and critic accuracy determines the reliability of KL-regularized, value-guided selection. Across simulated and real-world manipulation tasks, AnchorQ outperforms state-of-the-art post-training baselines in sample efficiency, delivering a 37% improvement in real-world success rate and 97% higher simulated task throughput under the same interaction budget. Additional visualizations are available on our website https://anchorq-rl.com/.