Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators
Machine LearningArtificial Intelligence
Summary
The authors study how to motivate a learning agent to explore environments where outcomes are uncertain in different ways. They propose a new intrinsic reward that balances curiosity by focusing on the part of surprise the model can actually learn about, avoiding both too much randomness and overfitting noise. This approach helps the agent explore unknown situations and eventually settle into predictable ones without needing an explicit prediction model. They also provide theoretical guarantees to ensure the learning process is stable and reliable. Overall, their method works consistently across changing conditions by carefully measuring useful novelty.
Authors
Alireza Furutanpey, Schahram Dustdar
Abstract
Across environments with mixed sources of uncertainty, unsupervised reinforcement learning requires intrinsic motivation that does not precommit to a particular direction of surprise. Surprise minimization is scoped by design to ``unstable'' environments. Prediction-error curiosity rewards total expected surprise, including irreducible noise. Bandit or mixture switching between surprise-minimizing and surprise-maximizing rewards reintroduces non-stationarity by construction. We propose a single intrinsic reward, stationary within each window, derived from the novelty contribution of a preference-free Expected Free Energy objective, expressed in reward-maximization form. Our claim is that parameter information gain, the expected surprise of the next state minus its irreducible part, is the appropriate intrinsic signal in both high-entropy and low-entropy components of the state space. Maximizing it seeks exactly the surprise the model can explain away. In regions of unresolved dynamics, this epistemic term drives exploration. As dynamics become resolved, the epistemic term vanishes, while an aleatoric penalty favors lower-variance transitions, all without fitting an explicit next-state predictor. A pseudocount supplies epistemic value, a probe-based penalty captures aleatoric variance, and a short-horizon gate protects informative successors. A window-based freeze of all reward-defining objects yields a stationary Bellman operator, explicit bounds on learning targets, and a conditional uniform-concentration result for the nonparametric estimators under mixing, smoothness, bandwidth, and capacity assumptions. In active-inference terms, the agent is preference-free where novelty is retained, standard likelihood ambiguity vanishes under full observability, a nonstandard transition-entropy penalty is added, and surprise minimization emerges in resolved regions of the state space.