Policy evaluation method improves risk estimation accuracy in rare failures
Tail-Influence Sampling for CVaR Policy Evaluation
Machine Learning
Summary
Sometimes, policies that seem good on average can still fail badly in rare cases. The authors study how to better estimate the risk of these rare failures by focusing on the lower tail of possible outcomes. They propose a method called Tail-Influence Sampling that spends evaluation effort where uncertainty most affects risk estimation, improving accuracy while using the same resources. Their method is tested on simulated environments and shows significant error reductions compared to common approaches.
What this means in practice
- •For reinforcement learning engineers: Allocate simulation queries to estimate policy tail risks more accurately within fixed evaluation budgets.
- •For automated review system developers: Improve error estimation in language model review workflows by focusing evaluation on risk-sensitive components.
Authors
Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer, Rasul Tutunov, Haitham Bou-Ammar
Abstract
Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can require many costly rollouts. When different conditional components of a stochastic workflow can be queried separately, we ask how to allocate a fixed evaluation budget to estimate a fixed policy's CVaR most accurately. We derive a tail influence for each queryable conditional law that aggregates how its uncertainty affects CVaR across every Bellman reuse. Its variance yields the fixed-design efficiency bound and the oracle Neyman allocation. Tail-Influence Sampling (TIS) estimates these influence scales from a pilot model and reallocates fresh queries toward kernels that matter most for the tail; a visitation-anchored variant protects against pilot underallocation. Under fixed dimension and a positive quantile margin, TIS attains oracle asymptotic variance and first-order MSE including pilot cost, while the anchored variant is within a factor two of the oracle. We also characterize an exact-grid regime in which tail- and mean-optimal allocations coincide. On CliffWalking, TIS reduces MSE by 41% versus learned occupancy and 76% versus complete rollouts at the same charged transition budget. In frozen language-model review workflows, anchored TIS beats an equally regularized mean-influence blend in 23 of 24 MMLU-Pro settings and reaches 2.4-3.4$\times$ lower MSE than rollouts on six-call FinQA reviews.