New method greatly improves estimation of extremely rare event risks
Rare Event Estimation via Iterative Unalignment
Machine LearningArtificial Intelligence
Summary
Sometimes computers or robots make rare mistakes that can cause big problems. Figuring out how often these rare mistakes happen is really hard because they involve very unlikely events in many possible sequences of actions. The authors developed a new way to estimate these probabilities much more efficiently by tweaking the way the system tries different actions, using a technique based on adjusting model weights and gradient searches. They tested this method on very large models and found it can estimate extremely rare events with much less computation than traditional methods.
What this means in practice
- •For autonomous vehicle developers: Estimate extremely rare failure event probabilities in self-driving car decision systems to improve safety evaluation.
- •For robotics safety engineers: Better quantify the risk of rare catastrophic action sequences in robots operating independently in complex environments.
Authors
Hanming Yang, Daksh Mittal, Jing Dong, Hongseok Namkoong
Abstract
As agents are deployed with increased autonomy, even extremely rare events along their stochastic output trajectories can occur and prove catastrophic. Safe deployment therefore does not depend on whether these events can occur, but on how often they might. We study the problem of estimating the probability of rare events that arise from stochastic variation in the agent's own actions. Estimating this type of risk requires searching over the combinatorially vast space of trajectories. Naive Monte Carlo is computationally prohibitive in this regime, and constructing effective importance sampling (IS) proposals requires coordinated changes to a context-dependent chain of conditional distributions. We develop a new IS method that perturbs the original model's weights to construct the proposal. The proposal is itself a differentiably parameterized language model, enabling gradient-based search over weight space. We formulate an objective that combines a differentiable surrogate for event amplification and an adaptive regularization scheme that dynamically balances amplification against estimator stability. We evaluate our approach on $\sim$120M and $\sim$2.6B models across three event families spanning 300+ rare events as rare as $10^{-9}$, with reference probabilities computed with $<10\%$ relative standard error. In our most verifiable settings, we observe that our IS estimator achieves over $800\times$ compute-weighted efficiency gains over naive Monte Carlo for events with probabilities lower than $10^{-7}$. Our implementation is available at https://github.com/namkoong-lab/iterative-unalignment.