Efficient exploration helps agents learn without external rewards
Efficient Exploration Is Enough
Machine LearningArtificial Intelligence
Summary
Exploring an environment is usually driven by getting rewards or completing tasks, but this paper looks at what happens when agents explore just to learn, without any outside rewards. The authors define efficient exploration as gathering useful experience that helps the agent learn to predict and adapt in new situations. They show that agents who focus on this kind of learning naturally start by exploring easier and more informative parts first, leading to a gradual buildup of more complex behavior. This suggests that simply trying to learn effectively can be enough for agents to develop sophisticated actions on their own.
efficient explorationintrinsic motivationgeneralizationpredictionreinforcement learningagentenvironmentcurriculum learningopen-ended learning
Authors
Mikel Malagón, Jon Vadillo, Josu Ceberio, Michael Bowling, Jose A. Lozano
Abstract
This work introduces an alternative view of efficient exploration and studies its theoretical and empirical implications in the absence of extrinsic rewards. Specifically, we define efficient explorers as agents that prioritize generating generalizable experience, i.e., data that supports learning models capable of predicting and adapting across the environment. This allows us to analyze efficient exploration through the lens of prediction and generalization. Theoretically, we demonstrate that optimally efficient explorers naturally schedule their trajectories to visit the most informative and learnable regions first. Empirically, we show that optimizing for these agents gives rise to an automatic curriculum of progressively more complex behaviors, even in relatively simple environments. These results indicate that pursuing this purely intrinsic objective alone is enough to drive the emergence of highly sophisticated behaviors. We believe that this new framework provides a principled mechanism by which agent-environment systems may sustain an open-ended process of increasingly complex behavior without external rewards, tasks, or objectives.