Streaming deep reinforcement learning helps robots adapt to changes online
An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics
Robotics
Summary
Robots often struggle when faced with new situations they weren't trained for, which can make them perform poorly. The authors show that if a robot first learns from a large amount of data, it can then improve itself continuously by learning directly from new experiences it encounters in real time. This approach helped walking robots adapt quickly to different changes and also partly helped robot arms, though with some limits. This research suggests ways to make robots better at learning and adjusting throughout their working life.
What this means in practice
- •For robotics engineers: Create robots that improve their movement policies continuously by learning from new real-time experiences after initial training.
- •For automation system designers: Develop robotic arms that adjust their control strategies online to unexpected task changes with partial success and stability considerations.
Authors
Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang, Richard Xue, Daniel Rakita
Abstract
Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes. In contrast, biological learning occurs moment-to-moment via a stream of experience, unlike the predominantly batch-based and offline nature of deep learning. Although recent works show the feasibility of stream-based deep reinforcement learning, where updates use only the latest experience, none have shown it to be a viable continual learning framework for adapting robotic policies to unseen changes. In this paper, we present the first analysis of streaming deep reinforcement learning for adaptive continual learning in robotics. In particular, we show that, following an initial pretraining phase, streaming deep RL can enable a robot to successfully adapt to unforeseen changes to itself, its environment, or goals. Our primary experiments within quadruped locomotion demonstrate that a deep neural network robotic policy with certain optimizers and plasticity loss mitigation techniques can successfully leverage domain task knowledge from its pretraining to quickly adapt online to diverse changes via stream learning, outperforming batch-based on-policy methods and improving task success rates by up to 90% over the pretrained policy. Furthermore, we perform additional evaluations on robotic manipulation tasks to determine if our previous observations extend to different robotic morphologies and scenarios. Our results show that the successes observed in quadruped locomotion can be partially realized in manipulation with stability and performance limitations. We conclude with a discussion on the limitations of our work and its implications for the future of continual robot learning.