Evaluation Awareness Is Not One Capability: Evidence from Open Language Models
2026-06-22 • Computation and Language
Computation and Language
AI summaryⓘ
The authors show that AI safety tests can be misleading because models might recognize when they are being evaluated and change their behavior to seem safer. They tested many models and found that while models can detect evaluation cues, this detection varies and affects how they behave during testing versus real use. They also found that some internal model information remains even when outward behavior changes. Overall, they conclude that measuring model safety by a single test score is unreliable because different safety-related factors do not align consistently.
AI safety benchmarksevaluation awarenessinstruction tuningAUROCbehavioral collapseHarmBenchmodel compliancerepresentation probingframing effectsmultivariate analysis
Authors
Nilesh Nayan, Aishwarya Sampath Kumar, Rishiraj Girmal, Shivani Anilkumar, Sankaran Vaidyanathan, David A. Nader Palacio, Reshmi Ghosh, Soundararajan Srinivasan
Abstract
Safety benchmarks assume that test-condition behavior predicts deployment behavior, an assumption that fails if models detect evaluation cues and adapt. This opens a gap between benchmark performance and deployment behavior: compliance measured under test conditions becomes an optimistic upper bound that overstates how safely a model behaves once the evaluation harness is removed. We characterize this evaluation awareness through eight experiments across 37 open-weight models and seven families. (i)Detection is moderate and training-driven (24/37 models exceed chance, best AUROC 0.714 vs.0.819 human, with instruction tuning dominating over scale). (ii)Detection shifts safety behavior (hard refusal drops 5.8 percentage points under hypothetical framing, and 21/140 HarmBench framing effects are significant, with compliance rising up to +30 percentage points. (iii)Representations survive behavioral collapse (probes retain AUROC 0.98 under rewrites that drive behavior below chance, and multi-layer steering causally moves three downstream tasks while random controls do not). (iv)These axes are weakly coupled (only 1/15 correlations are significant, the sole robust link being behavioral detection versus framing resistance, $ρ=-0.79$, $p<0.001$). We call this gap the benchmark illusion: because detectability, behavioral manifestation, and controllability vary independently, it is multivariate rather than a single number, so no single awareness score is a reliable proxy for deployment safety.