Training tricky questions improves large language model safety vigilance
Beyond Routine Compliance: Cunning Data Cultivates Safety Vigilance in Large Language Models
Artificial IntelligenceMachine Learning
Summary
Some language AI models struggle to detect harmful requests when danger is hidden in seemingly harmless questions. The authors studied a way to make models more careful by training them with 'cunning questions'—puzzles or sneaky challenges that require deeper thinking. This training helps models spot hidden problems and trickier dangers better, improving their safety checks. After extra training, models became better at rejecting harmful instructions earlier in their reasoning.
What this means in practice
- •For ai safety engineers: Improve defenses against deceptive harmful inputs by incorporating cunning question training into safety pipelines.
- •For chatbot developers: Enhance chatbot reliability by reducing successful prompt injections that hide malicious intentions.
Authors
Youjia Wang, Lin Xu, Yang Sun, Yuxiao Lu, Chengfang Fang, Jie Shi
Abstract
Safety alignment teaches large language models (LLMs) to recognize harmful requests and reject risky instructions. Yet aligned models can fail when harmful intent is concealed within seemingly benign contexts. Robust safety therefore requires both knowledge of safety boundaries and \textbf{vigilance}: the ability to detect unusual premises, misleading reasoning, and latent risks beneath surface-level semantics. Vigilance requires models to scrutinize a request's underlying intent and assumptions before acting. To cultivate this capability, we introduce \textbf{cunning questions}, which are not necessarily safety-related but contain misleading premises, atypical reasoning, or subtle inconsistencies. We hypothesize that learning to look beyond such reasoning traps can transfer to safety-critical scenarios. Experiments show that Cunning training improves robustness to out-of-distribution jailbreak attacks and strengthens subsequent safety fine-tuning. Furthermore, augmenting an existing state-of-the-art safety alignment pipeline with Cunning establishes a new state of the art across our evaluated settings, reducing mean ASR across nine backbone--benchmark combinations from 17.40\% to 15.05\%. Trace analysis after matched safety fine-tuning suggests that safety judgments are more likely to govern responses before harmful planning begins. A conditional theoretical analysis further characterizes when invariance learned from cunning data can transfer to safety-related inputs. These findings suggest that cunning data can strengthen model vigilance and complement conventional safety alignment.