Task decomposition improves robot learning speed and success with obstacles
Enhancing Deep Reinforcement Learning for Robotic Motion Planning via Task Decomposition
Robotics
Summary
Robots often learn to do tasks by trial and error in computer simulations, but they struggle and learn slowly when obstacles are involved. The authors found a way to help robots learn faster and better by breaking tasks into smaller steps and by reusing some knowledge from similar tasks. They tested this approach in simulated robot tasks with obstacles and showed better success rates and faster learning. They also tried their best method on a real robot to show it can work outside simulation.
What this means in practice
- •For robotics engineers: Train robotic manipulation policies more efficiently in environments with obstacles by decomposing tasks and selectively transferring learned features.
- •For warehouse automation teams: Improve robot motion planning and obstacle navigation in cluttered warehouse settings for faster deployment of reliable pick-and-place and pushing tasks.
Authors
Yuanzhi He, Victor Romero-Cano, Jiayu Xu, Juan David Hernández, Gualtiero Colombo
Abstract
Training robotic manipulation policies in simulation is widely adopted due to safety and cost efficiency; however, tasks involving obstacles remain under-performing and often converge slowly under Deep Reinforcement Learning (DRL). To address this challenge, we propose a task decomposition framework for DRL that integrates Transfer Learning (TL) with Reward Redesign (RR) to improve learning efficiency and robustness in obstacle environments. The TL strategy selectively transfers only the first-layer critic parameters across structurally related tasks, since that layer captures general state--action representations while deeper layers encode reward- and obstacle-specific value geometry; RR decomposes tasks into staged sub-objectives to provide denser optimization signals. Experiments on the PyBullet Panda platform show consistent gains in obstacle settings. On Push with obstacle, our method achieves $0.83 \pm 0.05$, outperforming the strongest baseline (Truncated Quantile Critics (TQC), $0.77 \pm 0.06$) by approximately $7.8\%$. On Slide with obstacle, performance reaches $0.95 \pm 0.02$, exceeding Soft Actor-Critic (SAC) ($0.84 \pm 0.06$) by about $11\%$ success rates. In addition to higher final performance, forward transfer reduces convergence time by up to $70\%$ in obstacle tasks, and RR shortens sparse-reward Pick-and-Place training from roughly $0.75$M to $0.20$M steps ($\approx73\%$ reduction). The best policy is deployed on a real Franka Emika FR3 robot as a qualitative feasibility demonstration.