Multi horizon curiosity boosts exploration in physical control tasks
Multi-Horizon Future Discovery for Physics-Informed Exploration
Machine Learning
Summary
Exploring how physical systems work is hard because some important changes only appear after a long time. The authors studied whether understanding many possible futures at different time points helps explore better. They created a method called HTC2 that learns to recognize connections between states far in the future while considering physics principles. This method encourages agents to seek out parts of the environment that are less understood, leading to more effective exploration across various simulated physical tasks.
What this means in practice
- •For robotics engineers: Improve robot exploration strategies in environments with complex physical dynamics and delayed effects of actions by using multi-horizon curiosity-driven rewards.
- •For game ai developers: Create more effective game AI agents that discover new game states by modeling the long-term consequences of their actions with physics-informed curiosity.
Authors
Shuwei Wang, Qiang Guan, Brian Qiu, Haoxiang Huang, Haoxi Wang, Chenyu Li, Bo Xu, Sen Cui
Abstract
Exploration in physical control environments requires agents to identify actions whose consequences reveal previously unexplored aspects of the underlying dynamics. However, many intrinsic motivation methods rely on immediate prediction errors, state novelty, or local physical changes, which may overlook informative outcomes that emerge across extended temporal horizons. We investigate whether learning structured representations of reachable futures can provide a more effective basis for exploration. To this end, we introduce Hamiltonian Temporal Contrastive Curiosity (HTC$^2$), a framework that combines multi-horizon temporal representation learning with Hamiltonian phase-space structure. A horizon-conditioned contrastive critic learns temporal relationships from future-state pairs sampled within the same episode. The distance between positive-pair representations is then used as an intrinsic reward, encouraging the agent to explore futures that remain insufficiently organized in the learned representation. To incorporate information about physical dynamics, we additionally modulate this reward using bounded multi-step Hamiltonian variation, rather than directly rewarding changes in the Hamiltonian itself. We integrate HTC$^2$ with both Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), and evaluate it across six DeepMind Control Suite tasks. Comparisons with ICM, RIDE, E3B, Plan2Explore, and a single-step Hamiltonian curiosity baseline are complemented by state-coverage evaluations, component ablations, and phase-space diagnostics to characterize the contributions of temporal learning and physical structure to exploratory behavior.