Policy variance influences learning ability in reinforcement learning
Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning
Machine LearningRobotics
Summary
Deep reinforcement learning models sometimes stop learning effectively from new experiences, which is called Loss of Plasticity (LoP). The authors studied a popular learning method called Proximal Policy Optimization and found that the amount of randomness in decisions, controlled by a parameter called policy standard deviation, affects how much LoP happens. They used experiments and math to explain why this happens and showed that adjusting this randomness helps keep learning active. Their tests included both simulations and a real robot arm, showing the idea works in practice.
What this means in practice
- •For robotic system engineers: Improve adaptive learning in robots with continuous controls by tuning policy variability parameters to reduce learning stagnation.
- •For autonomous vehicle developers: Enhance stability and ongoing learning of driving policies in self-driving cars by managing exploration parameters linked to policy adaptation.
Authors
Andrea Vasco Grieco, Evelyn D'Elia, Edoardo Caldarelli, Raffaello Camoriano
Abstract
Deep reinforcement learning (RL) algorithms have been observed to lose the ability to adaptively learn from new data, a phenomenon termed Loss of Plasticity (LoP). While many metrics have been shown to correlate with it, the causal mechanisms and the geometric conditions from which it can arise have not been clearly identified. We analyze LoP in Proximal Policy Optimization with isotropic Gaussian policies, the state-of-the-art formulation for many continuous action spaces in RL. We report multiple metrics which characterize the effect that the policy standard deviation $σ$ has on LoP, and we also suggest related explanations for the effectiveness of widely adopted LoP mitigation techniques. We observe a significant effect of $σ$ on the extent of LoP, and accompany this with both theoretical and experimental supporting arguments. Our experimental validation consists of LoP benchmark evaluations against standard baselines as well as real-world policy deployment on a 7-DoF robotic arm.