Agents switch between exploring and goal seeking in uncertain environments
Belief-Based Maximum Occupancy Principle and Active Inference
Artificial Intelligence
Summary
The paper looks at how decision-making agents can act when they aren’t sure about their surroundings and want to explore or reach goals without relying on explicit rewards. The authors extend a known idea, the Maximum Occupancy Principle (MOP), to work when parts of the environment are hidden, allowing agents to use beliefs about unseen things to decide their actions. They show that MOP agents can flexibly switch between exploring and hunting for food depending on their energy and what they believe, unlike other agents that tend to stick close to known good spots. This work helps us understand how behavior that mixes curiosity and purpose can emerge.
What this means in practice
- •For robotics teams: create robots that adaptively switch between exploring unknown areas and pursuing known goals using belief-based planning in uncertain settings
- •For autonomous vehicle developers: improve autonomous navigation by enabling vehicles to balance exploration of uncertain routes with goal-directed driving using belief-driven decision models
Authors
Manolis Mylonas, Rubén Moreno Bote
Abstract
Intrinsic motivation plays a central role in adaptive and goal-directed behavior by conferring agents reward-independent objectives and biases useful to act in noisy and uncertain environments. Active Inference addresses the problem of acting in a partially observable environment through a principled framework for belief updating and action selection. A key component of Active Inference is the specification of prior preferences, which shapes behavior by encoding desirable future outcomes. An intrinsic motivation approach called the Maximum Occupancy Principle (MOP) proposes that agents act so as to maximize occupancy over future paths of states and actions, with no preferences or epistemic targets. Despite its simple formulation, MOP gives rise to rich and adaptive behaviors that combine exploratory variability with goal-directed dynamics. In this work, we extend MOP to partially observable environments and introduce a Bellman reformulation of the Expected Free Energy for Active Inference, both incorporating belief-based inference over hidden states as part of the agent state. The Bellman formulation enables tractable offline computation via value iteration over the full belief-state space. We compare the resulting behaviors in a set of minimal experimental settings with uncertain food sources. We find that MOP agents switch between goal-directed (food seeking) behavior and exploration between different food sources, depending on their energy available and their belief state. In contrast, Active Inference agents mostly inhabit regions around a single food source, a strategy having both high pragmatic and epistemic value. We finally compare with Empowerment, which is shown to be qualitatively similar to Active Inference.