Black-box testing reveals major risks in autonomous AI agents
Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Artificial Intelligence
Summary
Many new AI agents act on their own and use outside tools, which can create security risks. The authors developed a way to test these AI agents without special access, by automatically generating tricky situations they might face. Their tests found that these AI systems often have serious risks related to rules, privacy, and unsafe behavior. This approach can help make AI agents safer before they are widely used.
What this means in practice
- •For ai safety teams: Evaluate deployed autonomous AI agents for hidden multi-step vulnerabilities without needing internal system details.
- •For security operation centers: Assess privacy and governance risks in multi-agent AI setups by running automated adversarial tests.
Authors
Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi
Abstract
Agentic systems are rapidly moving to production, where they read untrusted inputs, call tools with real permissions, and act autonomously, expanding the security surface beyond chat-only models. Yet standard evaluations remain single-turn and fail to capture multi-step agent vulnerabilities. We present a systematic black-box framework for risk-aware agent evaluation requiring only basic system descriptions. Our approach introduces: (1) a seven-domain taxonomy mapping observable behaviors to risk categories, (2) fully automated SAGE-RT red teaming producing 120 adversarial scenarios per domain, and (3) human-validated evaluation using LLM judges. Empirical validation across two agent architectures (CrewAI and AutoGen) with four base models reveals alarming patterns: 56.25\% average governance risk, 65\% privacy risk in multi-agent configurations, and agent behavior vulnerabilities reaching 85\%. Our black-box approach effectively identifies critical architectural vulnerabilities without privileged access, providing a scalable path toward safer agent deployments.