Limits of single-run safety monitors for large language models revealed
The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability
Machine LearningCryptography and Security
Summary
Checking the safety of large language models (LLMs) by looking at just one run or interaction is often not enough. The authors show that certain safety properties can only be reliably detected by comparing multiple runs, not by examining a single run. They measured how well current safety monitors work and found that they miss many signals especially when leakage between runs grows. The missing signals are not because the models lack capability but because of what and how we check safety. The authors suggest safety tests should be mechanically verified instead of relying on human or model audits.
What this means in practice
- •For ai safety engineers: Design safety monitoring tools for LLMs that incorporate multiple traces or runs to better detect subtle safety violations that single-run monitors miss.
- •For software testing teams: Create tests that mechanically verify hyperproperty-based safety benchmarks to avoid misleading evaluations that arise from human or model judgment biases.
Tested on one dataset.
Authors
Xin Xu
Abstract
Several properties safety monitors are asked to certify, among them cross-tenant noninterference, sandbagging and evaluation awareness, are 2-safety hyperproperties, witnessed only by two executions. The standard consequence is a binary impossibility: one trace cannot decide them. We replace the binary with a measurement. A tight bound puts the balanced accuracy of any single-trace monitor at $\tfrac12+\tfrac12\,TV(P_0,P_1)$, turning undecidability into a graded detectability frontier and defining an oversight gap: a monitor's shortfall below it. On a leak family with closed-form $TV$, nine LLM monitors are optimal at $TV=0$ but capture little signal as $TV$ grows; at $TV=1$, where a 20-line membership check scores $100\%$, they average $60.9\%$. That shortfall is mostly not capability: naming what to check closes $61\%$ of it while leaving the $TV=0$ control at chance. The same split runs through a $2{\times}2$ factorial: an imagined second run leaves monitors at chance ($50.4\%$) while the same rule on an executed second run reaches $90.0\%$, and a stored oracle without a comparison procedure yields only $68.2\%$. Information and procedure are each necessary and neither is capability. Under nondeterminism, replay tracks a closed-form $k$-replay curve only under the right projection, and a projection frontier shows the resulting dilemma is forced: narrow misses $98.6\%$ of off-channel leaks, broad flags $75.7\%$ of clean traffic, and attainable accuracy decays like $1/(qm)$ in the benign-variation rate and the channel count. Finally, two frontier LLM judges certified an earlier version of our own benchmark as sound while a sign test found a directional bias ($p=2.7\times10^{-5}$) that invalidated three of our findings. Construction validity for hyperproperty benchmarks should be proved mechanically, not audited by models.