Atomicrl improves robot learning for long task sequences
AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Robots often have trouble learning complex tasks made up of many smaller steps. The authors propose AtomicRL, a way to help robots learn by focusing on these smaller steps separately. This method improves how the robot figures out which parts of the task it’s doing wrong and where to explore more. Tests show AtomicRL helps robots do long and complicated tasks better both in simulations and the real world.
What this means in practice
- •For robotics engineers: Improve training of robot policies to handle complex tasks by assigning rewards to smaller subtasks and focusing on learning bottlenecks.
- •For industrial automation teams: Deploy robots capable of reliably completing long sequences of diverse manipulation tasks with improved learning efficiency and real-world transfer.$Commercial implications: Enables practical robot training pipelines that improve performance in manufacturing tasks requiring extended manipulation sequences, creating marketable automation solutions.
Authors
Likui Zhang, Zhe Zhu, Zhihao Zhan, Yibo Bai, Hanlin Wang, Dongwei Li, Bin Wang, Tianle Zhang, Xiaodan Liang, Liang Lin
Abstract
Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.