New method finds broadly useful goals for better AI exploration
Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery
Artificial IntelligenceMachine Learning
Summary
When AI explores large and complex environments, it needs shortcuts called options to make learning faster. Previous methods found goals that focused on every detail of a situation, which made these shortcuts useful only in very specific spots and led to too many confusing options. The authors created a new way to find simpler, more general goals by focusing on just a few important features. This helps the AI learn more quickly in tricky tasks like playing certain video games with few rewards.
What this means in practice
- •For game developers: Create AI agents that explore complex games more efficiently by learning generalizable goals rather than detailed state-specific ones.
- •For robotics teams: Develop robots that navigate and learn in large, sparse environments faster by using abstracted goals based on key features instead of exact states.
Authors
Akhil Bagaria, Anita De Mello Koch, George Konidaris
Abstract
Temporal abstraction via options can improve exploration in large environments. However, existing option discovery algorithms find subgoals that target all aspects of the state simultaneously. This state-reaching approach produces options that only apply in narrow regions of the state-space, eventually causing an explosion in the number of options that overwhelms the agent, and impedes progress on its primary task of reward maximization. We introduce an algorithm that instead identifies a small, relevant subset of features for each subgoal, yielding options that generalize broadly and accelerate exploration. Our approach learns abstract, transferrable options and achieves rapid exploration in three sparse-reward, image-based domains, including the Atari game MontezumasRevenge.