Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness
2026-08-10 • Computation and Language
Computation and Language
AI summaryⓘ
The authors explain that language models usually perform well in controlled tests but can struggle in real-world uses where prompts and rules are more complex. They create a new test called Decoding-Level Taboo that blocks common word choices during generation, forcing models to find alternative ways to respond. Their experiments show that larger models and those trained with extra instruction-following tend to handle these tricky situations better. This test can help check how reliable models are before they are used in actual applications.
large language modelsmodel evaluationlogit spacetoken maskinginstruction alignmentmodel robustnesssynthetic datasetsruntime safetymodel auditing
Authors
Tadanobu Chuyo Kamijo, Ori Rottenstreich, Javier Conde, Gonzalo Martínez, Pedro Reviriego
Abstract
Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance. To address this issue, we introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution. Evaluating Taboo across several open-weight model families reveals that off-path robustness is heavily influenced by both parameter scale and post-training instruction alignment, with robustness generally improving with model size and alignment. Beyond the results presented in this paper, Taboo provides a novel primitive for generating diverse synthetic datasets, stress-testing runtime safety guardrails, and auditing model reliability prior to real-world deployment.