Unified diffusion policy improves robot learning and exploration
CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance
RoboticsMachine Learning
Summary
Robots learning to perform tasks often need to try many actions to find good solutions, but common training methods can reduce their variety of behaviors. The authors propose CFG-Policy, a new approach that uses two separate policies: one to explore different actions broadly and another to focus on getting rewards. By combining these policies, the system keeps diversity in robot actions while also improving task success. This method works for both online training and when updating existing skills, and it was tested on robotic tasks in simulations.
What this means in practice
- •For robotics engineers: Train robotic controllers that maintain diverse behaviors while improving task success in simulation and real-world tasks.
- •For autonomous system developers: Enhance adaptive agents with stable exploration strategies when fine-tuning from pretrained policies to improve performance safely.
Authors
Shutong Ding, Zhaoyang Liu, Yimiao Zhou, Jingya Wang, Jiayuan Gu, Ye Shi
Abstract
Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks. Project page is at https://zhy-liu.github.io/cfg-policy-page.