Framework estimates rare severe behaviors in black box language models
Quantifying Behavioral Tails in Black-Box Language Models
Machine LearningComputation and Language
Summary
Measuring how often language models produce really bad or extreme results is hard because the range of possible inputs is huge. The authors made a tool called RareTrap that uses a simpler model to help explore input prompts in a smarter way, focusing on finding and estimating the chance of severe behaviors. This approach lets them efficiently test many language models, including recent ones like GPT-5.4 and Claude Sonnet 4.6, with only a few hundred tests. RareTrap helps those who build language models understand and prioritize fixing risky or harmful outputs.
What this means in practice
- •For language model developers: Quantify and prioritize fixing rare but severe undesirable behaviors in large language models under a common input distribution.
- •For ai safety engineers: Evaluate and mitigate risk of extreme resource consumption and other severe behaviors in deployed language models efficiently.
Authors
Elsayed Eshra, Ali Al-Lawati, Dongwon Lee, Suhang Wang
Abstract
We introduce RareTrap, a framework for estimating the probability of severe behaviors in black box large language models (LLMs). A key challenge for probability estimation is defining a tractable distribution over the input space. To accomplish that, RareTrap uses a surrogate LLM and constructs a geometry-aware mapping from a lower-dimensional latent reference space into its token-embedding space to induce an explicit and reproducible distribution over input prompts. A response-level performance function is utilized on the response to quantify behavior severity. This enables sequential rare event simulation that concentrates evaluations on progressively more severe behaviors while preserving probability under the induced prompt distribution, which would otherwise be prohibitive to measure. Across 10 open-weight and two frontier models (GPT-5.4 and Claude Sonnet 4.6), we find that RareTrap successfully induces severe resource consumption behaviors and computes their probability with as few as 200 evaluations. RareTrap provides model developers a principled approach for evaluating language models under a common distribution, and prioritizing alignment effort to improve safety and mitigate risks. Code is published online: https://anonymous.4open.science/r/rare_trap-D7F0.