Hierarchical reinforcement learning improves flexible task execution and policies
On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
Machine LearningArtificial Intelligence
Summary
Hierarchical reinforcement learning breaks tasks into smaller steps to explore solutions faster, but always finishing these steps can limit learning and flexibility. The authors explain two types of problems that happen: how the steps are executed and how the policy is learned. They develop a mathematical framework to analyze and improve both problems independently. Their experiments show that changing how the steps are executed and what learning targets are used can both improve results in different ways.
What this means in practice
- •For robotic system engineers: Design control policies that better balance completing subtasks and adapting execution for flexible robot behaviors.
- •For game ai developers: Improve game characters’ decision-making by separately optimizing action execution and learning objectives across hierarchical levels.
Authors
Bingyun Liu, Yuheng Jing
Abstract
Hierarchical reinforcement learning uses temporally extended subtasks for exploration, yet committing to their execution can restrict both deployment and policy learning. We identify and separate the resulting execution and policy suboptimality. Task and execution trees distinguish reward objectives from policy choices and decision interruption. A Unified Value Function for HRL and a four-stage Generalized Hierarchical Bellman Equation then support a common analysis of both losses. Under bounded rewards and uniform termination, we establish hierarchical policy and execution improvement results. With the remaining node policies fixed, task-subtree compatibility and node-policy optimality under the original execution mode establish when Markov execution is optimal. The resulting decomposition leads to independent execution choices for behavior, targets, and deployment. We instantiate this principle through execution improvement and one-stage or two-stage policy improvement at arbitrary hierarchy depth. Option-based and goal-conditioned experiments demonstrate complementary gains from changing execution and changing the learning target. Controlled stochastic environments show how these gains depend on stochastic transition strength and spatial structure. This framework makes execution design an explicit component of hierarchical policy optimization.