Intrinsic rewards sometimes fail to guide the best exploration
When Do Intrinsic Rewards Lead to Exploration?
Machine Learning
Summary
Exploration helps AI learn about new things, often by giving rewards based on curiosity signals like surprise or learning progress. The authors show that these internal rewards may not always push the AI to gather the most useful information. They create a way to measure how well exploration works by comparing different approaches on what could have been learned otherwise. Using a simple example, they find that many popular curiosity rewards lead to suboptimal exploration and explain when these rewards actually help. They also propose a new measure that better values improvement in exploration according to their criterion.
What this means in practice
- •For reinforcement learning engineers: Design better intrinsic reward functions for AI agents that more effectively explore environments by measuring informative experience gains.
- •For robotics developers: Improve autonomous robot exploration strategies by identifying when internal curiosity rewards fail to guide efficient data gathering.
Authors
Scott W. Viteri, Laura Gomezjurado Gonzalez, Clark Barrett
Abstract
Intrinsic rewards are designed to guide exploration in reinforcement learning by assigning value to an agent's experience, for example through prediction error or learning progress. However, maximizing these rewards need not produce the most informative experience available. We propose a formal criterion for exploration that compares policies by the counterfactual information they acquire: how well their histories can substitute for experience under alternative policies. We construct a single, simple environment in which specified count-based, prediction-error, empowerment, and information-gain objectives have maximizing policies that are Pareto-suboptimal at acquiring counterfactual information. We explain these failures and establish conditions under which existing intrinsic rewards successfully encourage optimal exploration. We also construct an objective that assigns a higher value whenever exploration strictly improves under our criterion.