TACS: Trajectory-Aware Candidate Selection for LLM Jailbreak Suffix Optimization
Computation and LanguageMachine Learning
Summary
The authors found that common methods for improving jailbreak prompts in language models focus too much on picking the best immediate candidate, which often leads to poor choices later on. They call this problem selection-stage reward hacking because the method tricks itself into favoring short-term wins that don't last. To fix this, the authors created a new method called \OURS that looks ahead to judge candidates based on their future potential, not just immediate results. Their tests show this approach leads to better and more stable jailbreak improvements. This work shows that choosing candidates wisely over time is important, not just making good candidates quickly.
Authors
Shiliang Xiao
Abstract
Gradient-based jailbreak suffix optimization methods typically update the suffix by retaining the candidate with the lowest current loss. We show that this seemingly natural design is fundamentally myopic: candidates that look better under the current-step proxy often fail to produce better jailbreak outcomes later in the search, revealing a form of selection-stage reward hacking. This suggests that candidate selection, rather than candidate generation alone, is a hidden bottleneck in suffix optimization. To address this issue, we propose \OURS{}, a trajectory-aware candidate selection framework for jailbreak suffix optimization. Instead of selecting candidates solely by their immediate loss, \OURS{} augments per-step evaluation with a trajectory-aware proxy and stabilizes selection with reference-policy regularization and a discriminator-estimated chi-squared correction, encouraging choices that remain effective beyond the current step. Experiments on HarmBench show that \OURS{} consistently outperforms strong baselines under the same search budget, substantially improving attack success rates while exhibiting more stable optimization behavior throughout the search. Our findings highlight that mitigating selection-stage reward hacking caused by myopic candidate selection is critical for improving jailbreak suffix optimization.