Selection-Aware Diagnostics for Chain-of-Thought Answer Hijacking
2026-06-03 • Cryptography and Security
Cryptography and SecurityComputers and Society
AI summaryⓘ
The authors investigate how certain types of reasoning in AI models can be manipulated to produce wrong final answers, even if the reasoning appears normal. They study this problem on math question datasets using different AI models, exploring how easily the wrong reasoning paths can be corrected or detected. Their results show partial recovery of correct answers when interventions are applied, but the success varies by model and task type. They also find that the source of clean data affects interpretations of these corrections. Overall, their work sheds light on the fragility and detection challenges of such reasoning hijacks.
chain-of-thoughtactivation patchingGSM8KMATH-500few-shot learninglocalizationinverse scalingbootstrappingWilson confidence intervaltransfer learning
Authors
Jianwei Tai
Abstract
We study a controlled numeric proxy for chain-of-thought (CoT) answer hijacking, motivated by attacks in which benign-looking reasoning steers a harmful final answer. CoT wrappers on GSM8K and MATH-500 flip final answers away from gold labels. Rather than treating activation patching as clean-trace restoration, we ask where hijacked trajectories are fragile and whether recovery depends on a same-problem clean source. Across Qwen2.5-7B and Llama3-8B on GSM8K few-shot, puzzle, and sycophant hijacks, three few-shot/puzzle cells pass confirmatory $K{=}1$ localization after Bonferroni correction. A selection-aware 50/50 band validation preserves held-out in-band minus out-of-band gaps of +32.6, +45.1, and +17.7 points for Qwen-puzzle, Llama3-fewshot, and Llama3-puzzle, while exact $\Lstar$ agreement is much less stable. Qwen-fewshot remains exploratory, and sycophant cells are temporal-diffuse under short patches. A BF16 Qwen-puzzle full-band sweep preserves the band signal ($n{=}30$, spread 0.33 at $K{=}1$, peak layer 20), supporting the conclusion that the band is not only an INT4 artifact. Fixed-hook GSM8K reruns preserve recovery in both primary puzzle cells: Qwen-puzzle recovers 47.0\% at $n{=}100$ (47/100; Wilson 95\% CI [37.5\%, 56.7\%]), while Llama3-puzzle recovers 39.0\% at $n{=}100$ (39/100; [30.0\%, 48.8\%]). Frozen transfer to MATH-500 recovers 26.0\% of qualified cases in the largest fixed-transfer run (13/50; Wilson 95\% CI [15.9\%, 39.6\%]). Source controls change the mechanism interpretation. Paired bootstraps give finite-sample non-separation between clean and random sources in Qwen-fewshot (+3.0 points, 95\% CI [-18.2,+27.3]) and Llama3-puzzle at expanded $n{=}60$ (clean--random -8.3 [-21.7,+5.0]), while Llama3-fewshot is content-mediated (+40.0 [+16.7,+60.0]).