Curriculum training improves automated jailbreak discovery in language models
Discovering Jailbreaks via Curriculum of Models
Machine LearningArtificial Intelligence
Summary
It can be very hard to get language models to reveal dangerous or harmful behavior because they usually refuse such prompts. The researchers found a way to teach a system to find these hidden weaknesses by starting with easier versions of the model that say yes more often, then gradually moving to the real, tougher model. This step-by-step approach helps the system discover bad behaviors more reliably than previous methods. It also works when looking for multiple behaviors at once, though some older methods can find more with enough training.
What this means in practice
- •For ai safety teams: Improve the detection of model vulnerabilities by training systems against easier-to-trick versions before testing on the deployed model.
- •For security auditing teams: Use curriculum-based training to more reliably find harmful prompt responses in large language models before deployment.
Authors
Michał Burzyński, Noah T. Rossi, Shomit Basu, Andy Arditi
Abstract
Automated jailbreak discovery aims to find prompts that elicit prohibited behaviors from safety-trained language models, helping developers identify vulnerabilities before deployment. One class of approaches uses reinforcement learning to train an investigator model to generate prompts that elicit a specified behavior from a target model. However, training directly against a safety-trained target is difficult because nearly all prompts are refused, leaving the investigator with a sparse reward signal. Prior work addresses this with the propensity bound (PRBO), a proxy reward estimated by sampling responses from a more compliant target, and then adjusting for how likely those responses would have been under the original target. We introduce Curriculum of Models (CoM), an alternative method that trains the investigator against a sequence of modified versions of the target, progressing gradually from a version that readily exhibits the specified behavior to the original target. For jailbreak discovery, training begins against a target version that rarely refuses harmful requests, and advances to the next, harder version once the investigator reliably elicits the behavior from the current one. We evaluate CoM on elicitation tasks involving CBRN materials, explosives, and illicit drugs, using two open-weight targets, gemma-4-31b-it and gpt-oss-120b. We compare against PRBO and direct training on the original target without a curriculum. When the investigator is trained to elicit a single behavior, CoM finds successful jailbreaks more reliably than either baseline. When trained on multiple behaviors simultaneously, the CoM-trained investigator can elicit held-out behaviors, though PRBO-trained investigators achieve higher success rates. Our results suggest that a simple curriculum over target models offers a promising approach for addressing sparse rewards in rare-behavior elicitation.