Structural jailbreak attacks on language models succeed but weaken in other languages
Structural Jailbreaks Generalize but Do Not Compound: A cross-provider and multilingual study of Involuntary In-Context Learning
Computation and LanguageCryptography and Security
Summary
Some language models can be tricked into harmful behavior through a technique called structural jailbreak, which makes them produce harmful content by treating bad requests like pattern tasks. This problem is much worse for financial abuse than other types of harmful content. The researchers found that this tricky behavior happens across different model providers and languages, but when using non-English languages, the models tend to produce less harmful or lower-quality outputs. So, the main risk is the English-language structural attack, not a combined weakness across languages.
What this means in practice
- •For ai safety engineers: Detect and mitigate structural jailbreak vulnerabilities in English-language language models to reduce harmful content generation.
- •For fraud detection teams: Focus on English-language financial abuse attacks in language models to prioritize risk reduction over multilingual concerns.
Authors
Tejasvi C. Addagada
Abstract
Aligned language models fail under two independent pressures: the structural jailbreak class recently formalized as Involuntary In-Context Learning (IICL), which reframes a harmful request as the final missing cell of a data-labeling task completed by pattern rather than judged as content; and the erosion of safety alignment outside English. A natural hypothesis is that these compound. We test it directly. Using a deterministic IICL operator and a StrongREJECT-style rubric judge, we red-team two Google Gemini models on two benchmarks, a 30 general-harm behaviours from HarmBench and 30 financial-abuse behaviours from FinProof, each under a single-shot baseline and under IICL in four languages (English, Spanish, Hindi, Arabic). First, IICL generalizes to a second provider and is worse in finance: it lifts attack success from <=6.7% to 80-90% on HarmBench and 97-100% on FinProof, an order of magnitude above the <=24% its introducing study reported on OpenAI's GPT-5.4. Second, against the hypothesis, forcing the IICL output into a non-English language does not stack the two weaknesses, it attenuates the attack. Eleven of twelve non-English conditions score below their English baseline (sign test, p~0.003), the lone exception a ceiling tie near 100%; on the stronger model's financial set Arabic collapses from 100% to 33%. We attribute this to a relevance curse: once structure has unlocked compliance, the models produce lower-quality harmful content in lower-resource languages, which a substance-grading judge scores as partial. The pattern replicates under an independent non-Google judge (Cohen's kappa=0.86, 377 paired verdicts), and 76.6% of non-English responses were verified in-language. Jailbreak vulnerabilities are therefore not additive; the dominant residual risk is the English structural attack, most acute for financial abuse, not a multilingual one.