AuraForge creates better security tests to train coding AI agents
AuraForge: Scaling Security Supervision for Training Coding Agents
Cryptography and SecurityComputation and LanguageComputers and Society
Summary
Writing computer programs that are both correct and secure is tricky, especially for AI coding assistants. The authors show that checking if code works isn’t enough to ensure it’s safe. They built AuraForge, a tool that automatically makes tests to catch security problems in code generated by AI. Using these tests, AI learns to write safer programs across different languages better than by relying on existing human-written security tests.
What this means in practice
- •For software development teams: Train AI coding assistants to generate more secure software by using automated, diverse security tests across multiple programming languages.
- •For security audit teams: Use AuraForge-generated security tests to better evaluate code from AI tools, reducing false positives and improving detection of vulnerabilities.
Authors
Danqing Wang, Songwen Zhao, Harsh Sharma, Jierui Wang, Andre Vicente Duarte, Ivan Bercovich, Lei Li
Abstract
Coding agents are now proficient enough to generate complex software applications from a single prompt. As their capabilities have grown, human oversight has increasingly shifted from line-by-line code review toward hands-off evaluation of outcomes. However, recent studies have shown that such a transition exposes a critical risk: functional correctness alone does not guarantee a secure implementation. Despite growing attention to code security, training safer coding agents remains challenging because reliable security supervision is difficult to obtain at scale from real-world repositories. We introduce AuraForge to synthesize and validate executable security tests for training secure coding agents. Our approach combines attack-oriented test synthesis, language-extensible task construction, and safeguards against reward hacking. Using AuraForge, we construct AuraGym, a multi-language and multi-CWE executable training gym: 679 executable feature-implementation tasks from 344 real-world repositories across Python, JavaScript, and TypeScript, covering 177 CWE categories. On the subset with human-written security tests, AuraForge produces about 3 times as many test cases on average and reduces the false-positive rate by 83.23%, allowing alternative secure implementations to receive correct supervision. Training Qwen3.5-4B with synthesized security tests gains larger improvements than human-written security tests (average 19.7 FuncPass and 6.2 SecPass vs. 14.9 FuncPass and 4.4 SecPass) on three languages. These results demonstrate that AuraForge provides more diverse and reliable security supervision to train secure coding agents.