Diffusion policy improves offline reinforcement learning actions
Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows
Machine LearningArtificial IntelligenceRobotics
Summary
When teaching computers to make decisions from past experiences without trying new actions, it's hard to pick the best moves if those moves weren't seen before. The authors propose a new method that starts with suggested actions and then improves them with a smart adjustment process that can handle many good choices. This method balances staying close to known actions while exploring better options and learns efficiently without complex computation. Tests show this approach works well on many benchmark tasks and improves further when the computer can practice online.
What this means in practice
- •For robotics engineers: Create robot controllers that improve actions by refining known behaviors without extensive online trial and error.
- •For autonomous vehicle teams: Enhance decision-making in self-driving systems using offline data combined with efficient policy refinement for safer navigation.
Authors
Junhyun Ha, Juho Lee, Byungwoo Park
Abstract
Diffusion and flow policies can model complex behaviors in offline reinforcement learning (RL). However, penalizing their KL divergence from the behavior policy can discourage actions having high critic values with low behavior density. Directly refining behavior proposals may be an alternative, yet Gaussian or deterministic editors limit expressiveness to represent multiple separated modes for the same proposal. In this work, we introduce Proposal-Conditioned Refinement Flows (PReFlow), a policy extraction method combining critic-based proposal selection with a conditional refinement flow. To optimize proposal selection and refinement together, we formulate a KL-regularized objective whose optimum induces a Gibbs policy over final actions under a Gaussian-smoothed behavior prior. The refinement flow can represent multiple high value modes, while a proposal-centered Gaussian reference regulates large action changes. This Gaussian reference further enables us to make use of simulation-free, closed form adjoint matching targets from sampled endpoints and critic gradients, yielding a single velocity regression loss without a backward adjoint solve. On 50 OGBench tasks, PReFlow achieves competitive offline performance and the highest aggregate score among the compared methods after online fine-tuning, reaching 91\% after 500K environment steps.