Measuring hidden harmful outputs in autoregressive language models
Refusal Is Not Robustness: Measuring Harmful Reachability in Autoregressive Language Models
Machine LearningLogic in Computer Science
Summary
Language models often refuse to produce harmful content, but they can still have a good chance of generating it. The authors introduce a way to look deeper into the model’s decisions, not just checking what it says out loud but how likely it is to say harmful things if it kept going. This method, called bounded continuation analysis, builds a tree of possible next words and measures how much risk of harm is hidden below a safe answer. It helps find risks that simple tests miss and guides better safety testing.
What this means in practice
- •For llm safety teams: Identify hidden risky behaviors under surface refusals to focus red-team efforts on higher hazard paths.
- •For ai content moderation teams: Assess models for hidden probabilities of harmful outputs beyond simple refusal checks to improve content filtering.
Authors
Parv Kapoor, Teresa Fortes, Claudia Mamede, Rui Abreu, Jonathan Aldrich, Eunsuk Kang
Abstract
LLM safety evaluations via jailbreaking typically index on attack success rate (ASR): whether a generated response violates safeguards. However, ASR only evaluates sampled outputs and provides little information on the underlying generation process. Autoregressive language models define distributions over possible outputs, and a sampled refusal can coexist with substantial probability mass on harmful generation paths. We study this hidden structure through bounded continuation analysis (BCA), which models autoregressive generation as a stochastic process over token prefixes and computes the finite-horizon probability that generation reaches a harmful continuation. BCA constructs a bounded continuation tree, labels terminal continuations with a semantic harm evaluator, and estimates harmful reachability using exact probabilistic model checking. BCA complements ASR by measuring how much harmful behavior remains accessible beneath a sampled safe response. This makes it useful for distinguishing models that look identical under measured ASR values, and allocating red-team effort toward regions of the generation process with greater hidden risk. Across four instruction-tuned LLMs and 100 JailbreakBench behaviors, we show that BCA effectively quantifies hidden harmful behavior and provides a useful signal to guide red teaming.