Diffusion models improve diverse motor skills for humanoid control
DSD: Learning Diverse and Reusable Motor Skills via Diffusion Skill Discovery
Machine LearningGraphics
Summary
Teaching robots or simulated characters to move in many useful and varied ways is hard because it’s difficult to measure how diverse their movements really are. The authors suggest using a special kind of AI model called a diffusion model to better estimate this diversity and guide learning. Their method helps create a wider range of motor skills that a character can reuse when performing new tasks. This makes the character more flexible and capable in different situations.
What this means in practice
- •For robotics engineers: Design control systems that reuse diverse motion patterns to adapt robots more efficiently to multiple tasks in simulation and potentially real-world settings.
- •For game developers: Create more varied and reusable character animations driven by learned motor skills that enable complex behaviors in game environments.
Authors
Sun Woo Kim, Xue Bin Peng
Abstract
Humans efficiently learn new tasks by reusing a rich repertoire of motor skills across different goals and contexts. A similar strategy can also be used to enable simulated characters to efficiently perform new tasks by leveraging reusable motor skills. To support a wide range of downstream tasks, the learned repertoire should be diverse, consisting of distinct behaviors as well as spatial and temporal variation within each behavior. A commonly used method for learning diverse skills is by maximizing the mutual information between skill latents and the states produced by a policy. The marginal state entropy promotes broad behavioral coverage, while the conditional entropy encourages consistent behaviors from each latent. However, directly estimating the marginal state entropy is intractable in high-dimensional control problems. Prior methods therefore rely on indirect latent-space approximations or coarse estimators of the state distribution. These approximations may not effectively promote broad coverage of the state space, resulting in skills with limited behavioral diversity and reduced utility for downstream tasks. In this work, we propose Diffusion Skill Discovery (DSD), a skill discovery method that uses a diffusion model to approximate the entropy gradient of the policy-induced state distribution through score matching. The resulting objective encourages the discovery of skills that produce a broader range of behaviors for high-dimensional humanoid control. The learned skills are reused in two downstream control settings: hierarchical control with a task-specific high-level policy and zero-shot control through latent selection from offline trajectories. Our experiments show that DSD discovers a broader repertoire of reusable motor skills than prior skill discovery methods, leading to the emergence of complex and agile behaviors that can be reused across downstream tasks.