Estimating agent empowerment from offline data to improve exploration

Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration

Machine Learning

Summary

Exploring new things is important for smart computers and robots to learn better. Usually, they learn from past experiences and then explore by trying new actions in the real world. The authors found a way to measure how many different future states an agent can reach (called empowerment) just from old stored data, without needing new trials. This helps the agent explore more effectively when it finally interacts with the environment, making learning faster and smarter.

What this means in practice

  • For robotics engineers: Improve robot learning speed by using prior data to guide exploration without extra environment trials.
  • For ai developers: Enhance goal-conditioned reinforcement learning agents to explore more effectively using empowerment estimated from offline experience.

Authors

Ishir Garg, Vivek Myers, Qiyang Li, Catherine Ji, Benjamin Eysenbach, Sergey Levine

Abstract

Large, unlabeled datasets of prior experience have become increasingly available for agentic tasks in domains such as robotics and coding. A common recipe for these tasks is to train a policy using the prior data, and then further improve it with online rollouts. But while much work has studied how to use both the offline and online data for policy extraction, far less work has been done to understand how the prior data can be used to efficiently \textit{explore} during the online learning phase. Previous work has attempted to formalize this idea by defining the \textit{empowerment}, a measure of how many future states an agent can realize; states with higher empowerment require more exploration as there are more futures to explore. However, existing methods for estimating empowerment require online policy rollouts in the environment. We propose the first method for estimating empowerment from unlabeled prior data, without any environment interaction. By jointly learning an empowerment-maximizing policy together with its discounted state-occupancy measure, our estimator can be trained on unlabeled trajectories generated from any unknown policy. We demonstrate that our empowerment estimator qualitatively gives intuitive empowerment measures and show that the pretrained estimator can be used to improve exploration in the online goal-conditioned reinforcement learning setting. Code is available at \url{https://github.com/ishirgarg/offline-empowerment}.