Summary
The paper studies whether artificial intelligence models can teach themselves to get better by using feedback from people. The researchers set up a system called RSI Arena where ten AI models independently trained themselves using limited computing power and budget, selecting data and training steps on their own. After a few days, the AI models produced useful assistants, with one nearly matching an existing top model in tests and human preference. The models showed some improvement when allowed to learn from comparisons, although gains were small and not always linked to what people preferred. This work explores how humans can guide AI agents to improve themselves safely and effectively over time.
What this means in practice
- •For machine learning engineers: Use autonomous training agents to improve language models based on human feedback within a fixed compute and budget framework.
- •For ai development teams: Deploy human-guided AI systems that iteratively refine themselves through feedback-informed training cycles for better user-facing assistants.
- •For customer support platform developers: Build AI-based assistants that can improve response quality by learning from user preferences and feedback autonomously.$Commercial implications: Enables creation of self-improving virtual assistants tailored to customer needs, enhancing service quality and reducing manual tuning.
Authors
Yue Huang, Yanbo Wang, Lichi Li, Yuanyuan Chen, Andreas Haupt, Xiaonan Luo, Zheyuan Liu, Jihye Choi, Felix Juefei-Xu, Xiangliang Zhang, Radha Poovendran, Misha Sra, Ying Liu, Ben Burtenshaw, Thomas Wolf, Alex Pentland, Zichen Chen, Zhangchen Xu
Abstract
Can AI agents autonomously train useful language models and improve them using feedback from users? We introduce RSI Arena, a two-stage study of autonomous post-training as a step toward recursive self-improvement. Ten frontier models act as training agents, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. During a six-day first stage, each agent has 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster, independently selecting data, implementing training procedures, and choosing checkpoints. We evaluate their models on 20 benchmarks and through blind human evaluations. All ten agents produce working assistants, with the strongest model, trained by GPT-6 Astra, closing a median of 76% of the base-to-Instruct gap across 12 capability benchmarks. Its human-preference rating is also close to NVIDIA Instruct after statistical adjustment for answer length and formatting. In Stage 2, each agent receives the comparison records and votes involving its Stage 1 model and is given three additional days for further training. Using NVIDIA Instruct as a common reference across the two evaluation rounds, we observe an improvement from Stage 1 to Stage 2 in the style-adjusted relative rating of eight of the ten models. We show that benchmark gains were modest, with no clear link to preference gains, while some agents encouraged longer responses and one model showed increased harmful compliance on StrongREJECT. RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development. It takes a step toward recursive self-improvement with human feedback (RSIHF), in which people help define what improvement means for the AI systems they use.