Asymmetric learning improves exploration in partial information tasks
Asymmetric On-Policy Distillation Learns In-Context Exploration
RoboticsArtificial IntelligenceMachine Learning
Summary
Making smart decisions when you can't see everything is hard because you first need to explore to learn important hidden information. The authors propose a method called ASTEROID where a smart 'teacher' helps a 'student' learn how to explore by showing the right moves based on what the student has observed so far. This approach helps the student learn to explore efficiently during tests without needing expensive trial-and-error learning. ASTEROID works well across many tasks, from robot manipulation to games, and even improves real-world robot success when vision is limited.
What this means in practice
- •For robotics engineers: Improve robot performance in tasks with hidden information by training policies that actively explore before acting.
- •For game ai developers: Create game-playing agents that explore unseen parts of the environment efficiently using student-teacher supervised learning.
Authors
Sriyash Poddar, Yanda Bao, Jacob Krantz, Matthew Chang, Xavier Puig, Roozbeh Mottaghi, Natasha Jaques, Abhishek Gupta
Abstract
Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively. Learning such information-gathering behaviors tabula rasa via online reinforcement learning is sample-inefficient, whereas offline imitation averages expert actions across hidden states and fails to learn exploration. In this work, we propose ASTEROID, a simple and effective student--teacher supervised learning framework that can learn test-time exploration. Although a privileged expert never needs to explore, we demonstrate that a history-conditioned student can learn to do so when trained with on-policy contexts: the student first collects on-policy context under partial observability, and the privileged expert provides action labels conditioned on that context. By distilling expert actions directly conditioned on the student's on-policy history, the student learns to systematically resolve partial observability, i.e., search while uncertain and exploit once the hidden state is revealed. This enables efficient test-time exploration (similar to Bayesian posterior sampling) without incurring the impractical training-data costs of reinforcement learning. Across a diverse range of eleven different environments, including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen environments, ASTEROID achieves ~2x higher returns under fixed exploration budgets, uses ~100x fewer interactions than RL baselines, and outperforms frontier models at exploration. Finally, our method transfers policies trained in simulation to vision-denied real-world robot manipulation, exhibiting efficient exploration and increasing success by +40% over standard distillation. Overall, it provides a simple yet surprisingly effective framework to learn efficient test-time exploration.