Agent systems improve their own training with co-design approach
VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement
Artificial IntelligenceComputation and Language
Summary
Making AI agents better requires improving multiple parts like their training data, the trainer, and how they use tools. The authors created VibeTraining-Bench, a way to test how well AI researchers can help agents improve all these parts together. Using this benchmark, they showed that current AI research agents can significantly boost agent performance but still need better ways to keep improving multiple parts step-by-step. This helps us understand how to build smarter AI that can improve itself over time.
What this means in practice
- •For software engineering teams: Improve automated agents that write and test code by iteratively refining their training data and evaluation setup for better coding assistance.
- •For machine learning engineers: Develop AI agents that can jointly optimize their training process and usage tools to enhance performance across complex tasks like reasoning and tool use.
Authors
Ruotian Peng, Yuanyi Ren, Shuaiqi Liu, Xin Guo, Fang Zhu, Yangyi Huang, Yue Wu, Zheng Wang, Yandong Wen, Zhihang Fu
Abstract
Recursive self-improvement (RSI) requires agents to diagnose system bottlenecks and coordinate improvements across training and inference. We refer to the agents' capability to jointly optimize training data, the trainer, and the inference harness as cross-component co-design. We introduce VibeTraining-Bench, a benchmark for evaluating this capability across tasks spanning terminal operation, software engineering, multi-turn tool use, and mathematical reasoning. Within each task, research agents start from an identical initial agent system under a fixed resource budget. Guided by validation feedback, they iteratively refine data, trainers, and inference harnesses, then select and submit a final agent system for evaluation on the test set. To prevent reward hacking, we enforce three safeguards: test data isolation, separate research and grading containers, and network restrictions for research agents. Our evaluation results show that frontier research agents substantially improve target agent systems. For example, on SWE-bench Verified, GPT-6-Astra raises the target agent system's performance from 32.9% to 53.7%, comparable to the human-developed instruct model's 55.4%. However, there is still room to improve their ability to co-design and iteratively refine data, trainer, and harness to achieve cumulative performance gains. We hope that VibeTraining-Bench will serve as a unified platform for advancing RSI, deepening our understanding of cross-component optimization for agent systems. Code and data are available at https://github.com/D2I-ai/VibeTrainingBench.