AI summaryⓘ
The authors analyze how systems that generate multiple possible outputs, filter them for safety using a learned model, and then choose the highest-reward safe output can still fail. They identify a problem called 'safety hacking,' where outputs that seem safe according to the model actually violate true safety rules, especially when many samples are generated. The authors provide mathematical bounds showing that increasing sample size can make safety hacking almost certain if unsafe outputs score high on reward within the model's accepted set. They also explore methods to control this issue but find that completely fixing the problem during inference isn’t straightforward. Experiments with toy and language models illustrate how unsafe outputs sneak through and get amplified, pointing out fundamental challenges in scaling up inference with learned safety checks.
inference-time pipelineslearned safety modelBest-of-N samplingsafety hackingreward maximizationfalse positivesχ² divergenceconstrained pessimistic samplingfeasible set contaminationreward-tail amplification
Authors
Akifumi Wachi, Takumi Tanabe, Youhei Akimoto
Abstract
Inference-time pipelines often sample multiple outputs, filter them with a learned safety model, and return the proxy-feasible output with the highest learned reward. We show that this composition creates a two-stage failure: an imperfect safety proxy first contaminates the feasible set with unsafe outputs, and reward maximization can then amplify this residual contamination. We define \emph{safety hacking} as selecting an output that passes the learned constraint but violates the true safety criterion. For constrained Best-of-$N$ sampling, we derive finite-$N$ bounds governed by the joint upper reward tails of safe and unsafe outputs within the proxy-feasible set. If unsafe-but-feasible outputs have the heavier tail, safety hacking becomes asymptotically certain as $N$ grows, even when false-positive mass and average safety- and reward-proxy errors are arbitrarily small. We also show that policies within a bounded $χ^2$ divergence from the proxy-feasible reference distribution admit an $N$-independent safety-hacking bound, and instantiate this general coverage-control principle with constrained pessimistic sampling. Coverage control limits amplification but cannot repair a contaminated feasible set: admitted unsafe outputs may still be favored, and regularized selection is not necessarily safer than constrained Best-of-$N$ for every reward proxy. Toy and language-model experiments characterize both contamination and its reward-tail amplification, which exposes an inherent difficulty in inference-time scaling with learned safety models.