Geometric regularization boosts online reinforcement learning success rates
GSBR: Geometric Support Behavior Regularization for Online RL
RoboticsMachine Learning
Summary
Teaching robots to improve their actions by learning from a pre-trained model can be tricky because sticking too close to the model limits improvement, while straying too far can lead to mistakes. The authors found a new way to gently guide learning by only penalizing actions that differ too much from the model’s typical actions, allowing room for better choices. Their method, called Geometric Support Behavior Regularization (GSBR), lets robots learn more effectively and reliably. Testing on many tasks showed it greatly increased success rates compared to other methods.
What this means in practice
- •For robotics engineers: Improve robot learning stability and speed by guiding policy updates using the new geometric support distance penalty.
- •For industrial automation teams: Deploy robotic systems that learn tasks more reliably by balancing adherence to known behaviors with room for improvement.
Authors
Charles Xu, Jacob Berg, Liyiming Ke, Sergey Levine
Abstract
Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets -- capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy's support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code at https://charlesxu0124.github.io/gsbr/ .