Framework measures how well self-improving ai handles new challenges
SelfImprove-Bench: Evaluating Generalization in Self-Improving Agents
Artificial IntelligenceComputation and LanguageMachine LearningMultiagent Systems
Summary
When computer programs try to get better at tasks by improving themselves, it’s hard to tell whether they actually learned new skills or just got better at specifics they already saw. The authors created SelfImprove-Bench, a testing setup for these programs that focuses on exploring, planning, and reasoning skills in game-like tasks. It checks how well improvements made during training carry over to new, unseen challenges. They found that current methods don’t always consistently improve across more general situations.
What this means in practice
- •For ai development teams: Evaluate how well self-improving AI agents perform on new tasks beyond their training scenarios using a standardized benchmark.
- •For game ai programmers: Test planning and reasoning improvements in AI agents with controllable game scenarios that scale in difficulty and variety.
Authors
Bangrui Xu, Jie Yang, Sidi Chen, Xuanhe Zhou, Jiantao Qiu, Qianyu Yang, Zijie Wang, Fan Wu, Zhoufutu Wen
Abstract
Recursively self-improving (RSI) systems are increasingly viewed as a promising means of accelerating progress toward artificial general intelligence. They aim to autonomously improve their capabilities by refining their working pipelines and recursively improving the strategies that guide subsequent refinements, with gains generalizing to unseen scenarios.However, existing evaluations lack reliable metrics to distinguish capability gains from scenario fitting, sustainable and extensible protocols for unseen tasks and difficulties, and fine-grained attribution of improvement to evolution traces. To this end, we propose SelfImprove-Bench,an RSI evaluation framework that organizes tasks into three primary capabilities (i.e., Exploring, Planning, Reasoning) as scenarios for guiding model evolution. We instantiated the tasks on games for controllability and reproducibility, and pairs them with scalable generation and orchestration driven by task templates and difficulty coefficients. Beyond task-level scores, SelfImprove-Bench organizes evaluation into generalization ladder levels G0 to G4 that measure how far gains transfer from training conditions. Experiments reveal that the evolutionary gains of existing RSI methods are inconsistent across generalization levels.