Unsupervised reinforcement learning discovers basic robot skills autonomously
Bellman Meets Lyapunov: Unsupervised Reinforcement Learning via Mastering Chaos
Machine Learning
Summary
Reinforcement learning usually needs carefully designed rewards by experts for each new task. The authors introduce a way for agents to teach themselves useful behaviors without any human-designed rewards. Their method, called Forward CIP, relies only on how the system changes over time, so it doesn't need expert knowledge to set goals. Agents trained this way can naturally learn important skills like balancing before moving onto more complex tasks. Adding a simple speed goal then helps them learn walking and running gaits without extra engineering.
What this means in practice
- •For robotics engineers: Train robots to autonomously discover fundamental control behaviors like balancing and locomotion without handcrafted rewards.
- •For game ai developers: Enable game characters to learn coordinated movement patterns from environment interaction without pre-set reward signals.
Authors
Tristan Shah, Wooyoung Chung, Volodomyr Makarenko, Juan Wachs, Stas Tiomkin
Abstract
Reinforcement learning (RL) is a powerful paradigm for training agents, yet its success rests on domain expertise of human engineers who design informative reward signals for every new task. Unsupervised RL aims to reduce this engineering with intrinsic motivation (IM): reward signals that emerge from the agent environment interaction itself. Existing IM objectives, however, involve the selection of information variables, which re-introduces domain expertise the field has sought to eliminate. We introduce Forward CIP (F-CIP), an RL-native formulation of the Controllable Information Production (CIP) objective, which is defined by the system's dynamics alone and requires no such selection. We prove that F-CIP is compatible with RL and demonstrate its effectiveness with existing algorithms. Training agents with F-CIP results in unsupervised discovery of primitive behaviors such as balancing and maintaining controllability, which are essential for more complex robot behaviors. Paired with a simple forward-velocity reward, our method produces coordinated gaits such as hopping and running which otherwise require reward engineering to learn.