Hierarchical rl models improve decision making under uncertainty
A note on goal-based hierarchical RL
Artificial Intelligence
Summary
Reinforcement learning helps machines make decisions by choosing goals and actions. Previous work made these choices either fully visible or given externally. This paper combines these ideas, using a model that handles hidden information and lets the machine decide its own goals and when to finish them. This approach could better reflect complex decision-making in uncertain situations.
What this means in practice
- •For robotics engineers: Design robots that can set and complete goals autonomously in uncertain environments by integrating hidden state planning with goal decision mechanisms.
- •For game ai developers: Build more adaptive game characters that internally manage goals and actions without full environment knowledge using hierarchical models.
A position paper. It proposes an approach and reports no results.
Authors
Kevin Murphy
Abstract
The agent-centric general value function (ACGVF) construction of \citet{tasse2026goal} lets the agent make two decisions that are normally imposed by the environment or agent designer: which goal to pursue and when to declare a goal as finished (in addition to choosing the action). This is a very general framework that subsumes almost all prior work on reinforcement learning, control and planning, as well as more general formalisms proposed in the cognitive sciences. However, it assumes the environment is fully observed, i.e., that the observation is a sufficient statistic. In \citet{murphy2025rl}, a general agent design was proposed where the policy is based on an internal belief state $z_t$ and an internal goal; however, the goals were assumed to be externally provided. In this note, we unify and extend these two approaches using the formalism of hierarchical hidden Markov models (HHMM) \citep{murphy2001hhmm}.