Efficient explorers create complex behaviors without external rewards
Efficient Exploration Is Enough
Machine LearningArtificial Intelligence
Summary
Finding interesting ways to explore an environment can be done without needing rewards or tasks to guide an agent. The authors define efficient exploration as gathering experiences that help an agent learn how to predict and adapt to different parts of the environment. They show theoretically that the best explorers choose their actions to first learn the most useful and understandable parts of the environment. Their experiments demonstrate that this approach naturally leads to increasingly complex behaviors, even in simple settings. This means agents can develop sophisticated skills just by trying to learn effectively, without any external goals.
What this means in practice
- •For robotics developers: Design robots that autonomously learn complex tasks by focusing on self-guided exploration without needing explicit rewards or user instructions.
- •For game ai engineers: Implement game characters that gradually develop sophisticated behaviors through intrinsic objectives, enabling richer non-player character interactions.
Authors
Mikel Malagón, Jon Vadillo, Josu Ceberio, Michael Bowling, Jose A. Lozano
Abstract
This work introduces an alternative view of efficient exploration and studies its theoretical and empirical implications in the absence of extrinsic rewards. Specifically, we define efficient explorers as agents that prioritize generating generalizable experience, i.e., data that supports learning models capable of predicting and adapting across the environment. This allows us to analyze efficient exploration through the lens of prediction and generalization. Theoretically, we demonstrate that optimally efficient explorers naturally schedule their trajectories to visit the most informative and learnable regions first. Empirically, we show that optimizing for these agents gives rise to an automatic curriculum of progressively more complex behaviors, even in relatively simple environments. These results indicate that pursuing this purely intrinsic objective alone is enough to drive the emergence of highly sophisticated behaviors. We believe that this new framework provides a principled mechanism by which agent-environment systems may sustain an open-ended process of increasingly complex behavior without external rewards, tasks, or objectives.