Off-policy reinforcement learning improves robot bimanual task performance

Off-Policy RL with Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge

RoboticsComputer Vision and Pattern RecognitionMachine Learning

Summary

Teaching robots to do tasks with both hands needs precise and safe coordination. The researchers improved robot learning by tuning how the robot guesses its best moves while learning from past experiences, even reusing its own mistakes to get better. Their method helped the robot complete complex tasks almost as well as human demonstrations. The approach combines a general learning method with specific corrections to reach high scores on challenging robot tasks.

What this means in practice

  • For robotics engineers: Train robot arms to perform coordinated tasks more safely and efficiently by tuning inference during off-policy reinforcement learning.
  • For robot control developers: Improve robot task success rates by augmenting training data with recovery episodes and high-quality generated demonstrations.

Authors

Ilia Larchenko

Abstract

The CoRL 2026 RoboEval challenge has eight bimanual tasks, and it ranks a policy not only on whether it succeeds but on how: every stage of the task, done safely, efficiently and with both arms in coordination, scored against human demonstrations. My solution has two parts. The generic part, carried over from my LeHome challenge solution, is off-policy RL on a $π_{0.5}$-based policy that is also its own value function. It is trained with AWR and RECAP on advantages of the exact official score, in a loop where every rollout is at the same time a Thompson-sampling trial of about 20 inference parameters. Out of the box, it took the policy to 0.85 RDS on held-out seeds in two days (and it was still very far from convergence). The competition-specific part adds data that local exploration cannot find: generated demonstrations that reach the full reward, and recovery episodes that start from the policy's own failures. The submitted policy reaches 0.990 RDS on held-out seeds and 0.987 on the official seeds.