AI safety challenges rise with recursive self-improving systems
Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation
Artificial Intelligence
Summary
As artificial intelligence systems start improving themselves, keeping them safe becomes harder. The authors highlight how safety can change or weaken over time in these evolving systems through issues like mistakes piling up or goals shifting. They create a way to classify different kinds of risks and explain how to spot and measure these dangers. They also suggest rules for managing changes and recovering from problems to keep AI safe as it improves itself again and again.
What this means in practice
- •For ai safety teams: Use the taxonomy and risk evaluation methods to monitor and manage safety risks in AI systems that update themselves repeatedly.
- •For autonomous system developers: Implement governance principles to control modifications and handle failures in evolving AI agents that learn and improve independently.
A position paper. It proposes an approach and reports no results.
Authors
Chang Gong, Jingping Bi, Di Yao, Xinjian Liang, Chao Xiang, Ruijie Guo
Abstract
Artificial intelligence is advancing rapidly, with increasingly capable systems taking larger roles in reasoning, decision-making, scientific discovery, and autonomous development. As AI begins to participate in its own improvement, from model training and experience accumulation to agent evolution and automated AI development, the prospect of recursive self-improvement (RSI) is becoming increasingly relevant. This transition raises a fundamental safety question: how can safety be maintained when the system, its accumulated experience, and even the process producing its successors continue to change? We introduce Evolutionary Safety as a perspective for studying safety under persistent and recursive self-improvement. It concerns not only whether an AI system is safe at a particular moment, but how safety properties change, persist, accumulate, and propagate throughout evolution. We characterize recurring manifestations, including intent drift, error accumulation, experience contamination, safety-property erosion, evaluator drift, and risk propagation. We then develop a taxonomy spanning persistent agent state, model state, evaluation and environmental feedback, computational substrate, and meta-level update mechanisms. Building on this taxonomy, we examine how evolutionary risks can be discovered and evaluated across states, updates, trajectories, and lineages, and derive governance principles for modification, selection, authorization, provenance, and recovery. Finally, we outline open problems toward maintaining safety guarantees as AI systems become increasingly persistent, adaptive, and recursively self-improving. Project resources and proposed evaluation systems are available at https://chaunceykung.github.io/evolutionary-safety-rsi.