Target model reviews failures best but another proposes attacks better
The Target Agent Is Its Own Best Critic but a Poor Proposer: Self-Improved Red Teaming
Cryptography and SecurityArtificial Intelligence
Summary
This paper looks at how automatic testing of AI systems using attacks can be improved. The authors show that when the AI system itself explains why an attack failed, it gives better reasons than a separate model would. However, if the system tries to come up with the next attack itself, it tends to avoid doing so effectively. They suggest a two-part approach where the AI system critiques attacks internally while a separate attacker model makes new attack proposals, which works better in testing different AI tasks.
What this means in practice
- •For security testing teams: Improve vulnerability scans by separating critique and attack roles across different AI sessions for stronger attack success.
- •For ai first responders: Use split-role red teaming to better identify and fix AI system weaknesses in conversational and workflow applications.
Authors
Mintong Kang, Stephen Rawls, Melissa Kazemi Rad, Jian Ni, Bo Li
Abstract
Automatic red teaming increasingly relies on large language models (LLMs) to iteratively diagnose failed attacks and propose improved adversarial prompts or injections. Although prior work strengthens this loop through prompt engineering and search-harness design, the mechanisms underlying its two core roles, criticism and attack proposal, has not been explored. To further investigate the core roles and therefore improve red teaming effectiveness, we disentangle two design choices: whether criticism is generated by the target session itself (self-criticism) or by a separately instantiated critic (external criticism), and whether that criticism is translated into the next attack by the target session (self-proposal) or by a separate attacker (external proposal). Through our analysis, we find that self-criticism is more effective because it is grounded in the target's live execution context, including observations, constraints, and available reasoning; removing reasoning reduces downstream attack success by 6%-17%. In contrast, self-proposal is substantially less effective because awareness of the adversarial objective induces explicit refusal and non-refusal sandbagging. These findings motivate a role-separated red-teaming approach: the target session diagnoses why an attack failed, while a separately instantiated attacker generates the next proposal. On DTAP-BENCH, spanning Telecom, Workflow, CRM, and Code domains, under both direct misuse and indirect threat models, as well as multiple agent frameworks, this configuration achieves 63.2% overall attack success rate (ASR), much higher than baselines. Our results show that effective automatic red teaming depends not only on model capability and search strategy, but also on how execution context and adversarial responsibility are distributed across model sessions.