Safe self-improving AI faces limits on reliable progress and evaluation
Safe Harness Self-Evolution: A Theoretical Analysis of Feasibility and Limits
Artificial Intelligence
Summary
This paper studies how AI systems can safely improve themselves by changing the way they work without altering their main brain or language model. The authors analyze when such self-improvements are possible and how to make sure new changes are safe and actually helpful based on limited feedback. They find that it can be hard to find good improvements even if they exist, and checking for real improvements can become very costly as the system gets better. Their work helps understand the challenges and limits of building AI systems that can evolve safely over time.
harness self-evolutionlanguage modelexpected rewardmodification generationcertificationtask feedbacksafe adoptionevaluation costprobability boundsself-improving AI
Authors
Qianshu Cai, Yonggang Zhang, Jun Nie, Maohao Ran, Huajiang Zheng, Jun Song, Xinmei Tian, Yike Guo, Wei Xue
Abstract
Harness self-evolution is the process by which an agent modifies its prompts, tools, code, or orchestration in response to task feedback while keeping the underlying language model frozen, with changes persisting across subsequent tasks. We provide a systematic theoretical analysis of the feasibility and limits of safe harness self-evolution, connecting modification generation, finite-data certification and selection, safe adoption, and behavior after an update. Under a fixed user-task distribution, we establish conditions guaranteeing overall expected-reward improvement while controlling changes on retained tasks, characterize the probability of generating qualified modifications, and derive finite-data bounds for safe selection and adoption. Our analysis shows that generation and certification impose distinct constraints: current task performance does not determine the probability of generating qualified modifications, and generating more candidates need not improve the guarantee of a successful update when evaluation is limiting. Stagnation may therefore arise even when improvement opportunities remain. We further show that worst-case evaluation cost for recognizing genuine improvements diverges as expected reward approaches its upper bound. Across successive updates, certified improvement guarantees accumulate over a finite run, but a successful update does not by itself guarantee that further improvement remains possible. These results provide a basis for diagnosing bottlenecks and designing safer self-evolution mechanisms.