Adaptive red teaming finds more weaknesses in large language models
CART: Closed-Loop Adaptive Red Teaming for Large Language Models
Artificial Intelligence
Summary
Testing large language models usually involves using a fixed set of questions, which only finds known problems. This paper presents CART, a method that learns from each test to decide what to check next, helping find more and different issues. CART separates the roles of creating tests, the model being tested, and judging results, making the process clearer and easier to improve. The authors show that CART finds more serious problems than fixed tests and works even for complex agents using tools.
What this means in practice
- •For ai safety teams: Identify unknown and evolving weaknesses in language models to improve safety measures before deployment.
- •For software developers building ai tools: Continuously test AI-driven applications for new vulnerabilities uncovered by adaptive test strategies.
Authors
Dongdong Zhang, Tengchao Lv, Yilin Jia, Yuzhong Zhao, Yupan Huang, Wenshan Wu, Xiangyang Zhou, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Furu Wei
Abstract
Automated red teaming often replays a fixed set of prompts, which measures known risks but cannot learn from failures found during testing. We present CART (Closed-Loop Adaptive Red Teaming), a framework that uses each result to guide what it tests next. CART begins with broad risk coverage, follows weaknesses that emerge, keeps new probes diverse, and records the evidence and source of every finding. It separates the Challenger that creates tests, the Target being tested, which may be a text-only model or a bounded tool-using agent, and the Judge that evaluates the results, allowing these roles to be studied independently. Across three evaluation families (Frontier, JAH, and Agentic), CART discovers more failures and higher average risk than static seed replay for every Target with an available baseline. The gains extend to tool-mediated agent tests, suggesting that contextual adaptation can reveal weaknesses that direct prompt replay does not exercise. These results describe what the test policies discover, not how often failures occur in real deployments. We also find that Challenger-Judge choices affect the evidence uncovered, highlighting the need for role separation and independent review. Overall, CART turns red teaming from a one-time checklist into a continuous, adaptive, and auditable search for model and agent weaknesses.