Evaluating hierarchical decision policies from flat data cuts bias
Intra-Option Fitted Q-Evaluation: Evaluating Hierarchical Policies from Non-Hierarchical Data
Machine Learning
Summary
Estimating how well complex decision-making plans work without testing them in real life is hard, especially when these plans include several steps that last different amounts of time. The authors found that existing methods that try to evaluate such multi-step plans often need extra information or make errors that grow the longer the plan is. They developed a new technique called Intra-Option Fitted Q-Evaluation (IO-FQE) that can accurately assess these plans using ordinary data that doesn't label each step. Their method reduces errors and bias, demonstrating better performance on tasks like navigating mazes or solving puzzles using robots.
What this means in practice
- •For autonomous robotics teams: Evaluate complex multi-step robot control strategies accurately without requiring option-labeled data from prior runs.
- •For reinforcement learning engineers: Improve policy evaluation for hierarchical RL models deployed in long-horizon continuous control tasks by reducing bias and error.
Authors
Yunfu Deng, Josiah P. Hanna
Abstract
Off-policy evaluation (OPE) estimates the expected return of a target policy from previously collected data without additional environment interaction. While OPE methods for flat Markovian policies are well studied, little work has addressed evaluating hierarchical policies in which a high-level policy selects temporally extended options that generate primitive actions until a termination condition is met. The primary existing approach applies per-decision importance sampling at the option level, but requires option-annotated trajectories and exhibits variance that still grows with the horizon. For non-hierarchical policies, fitted Q-evaluation (FQE) typically achieves lower error than importance sampling by leveraging the classic Bellman equation for policy evaluation; however, a naive extension of FQE to hierarchical policies results in a biased policy value estimate. In this paper, we introduce Intra-Option Fitted Q-Evaluation (IO-FQE), which estimates a value function in the augmented state-option space; this approach enables OPE of hierarchical policies even when we lack annotations of what option was run in the data. We develop two continuous-action instantiations of IO-FQE and show on long-horizon hierarchical continuous control tasks (AntMaze navigation and OGBench Puzzle manipulation) that IO-FQE eliminates the bias incurred by a naive application of FQE to hierarchical policies and substantially lowers mean squared-error compared to option-level importance sampling.