Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
2026-07-10 • Robotics
RoboticsArtificial Intelligence
AI summaryⓘ
The authors study how accurately estimating the value of actions affects learning robot manipulation tasks from offline data and then improving online. They introduce Robo-ValueRL, a system that assesses how reliable these value estimates are, and uses that to guide training and improvement of robot policies. Their experiments show that more reliable value estimates lead to better robot performance and more efficient use of data. This approach helps robots perform complex tasks like precise chip insertion and block disassembly more successfully. The authors emphasize that focusing on value accuracy helps better use diverse robotic experience for training.
offline reinforcement learningonline reinforcement learningvalue functionpolicy optimizationrobotic manipulationpolicy pretrainingrollout trajectoriesvalue-guided learningbehavior cloningdata prioritization
Authors
Wenke Xia, Pei Ren, Wenbo Yu, Yizhuo Zhang, Jifan Li, Yixue Zhang, Yinuo Zhao, Qingyang Gao, Jianlong Fu, Jian Tang, Ji-Rong Wen, Zhengping Che, Di Hu
Abstract
Offline-to-online reinforcement learning is promising for generalizable robotic manipulation, yet its full-stack complexity obscures reproduction and diagnosis. Within such systems, value estimation plays a central role in prioritizing heterogeneous data for policy improvement. Despite its importance, the central question remains underexplored: how value-function reliability shapes policy optimization in offline-to-online reinforcement learning. To answer this question, we propose Robo-ValueRL, a unified framework that enables reliable value estimation and systematically traces its downstream effects on policy pretraining and online improvement. Concretely, Robo-ValueRL learns a history-conditioned value estimator and evaluates its reliability through global-progress and local-preference metrics. These resulting value estimates are propagated into quality-conditioned consistency-policy pretraining and a residual adaptation module on online rollouts, providing a unified testbed for analyzing how value reliability shapes downstream policy performance. Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, our extensive experiments show that downstream performance is strongly associated with value reliability. Reliable value functions provide better action-quality estimates, allowing value-guided offline RL to scale more effectively than quality-agnostic behavior cloning, and stabilize online improvement by prioritizing high-quality rollout data. Integrating reliable value guidance through offline pretraining with online improvement, our system achieves 86% success on millimeter-level precise chip insertion and 84% on generalizable block disassembly. We hope these findings highlight the importance of value-guided data utilization for effective policy improvement from heterogeneous robotic experience.