Limitations on automated self-improvement in AI systems
The Self-Improvement Problem
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
The paper studies whether AI systems that improve themselves can safely decide which improvements to accept. It finds that no perfect method exists to always accept only improvements without missing any, due to fundamental logical limits. However, by restricting the situation—like limiting how far the system looks ahead—some safe methods become possible. Still, even these safe methods have subtle problems that can mislead the system. The authors’ work highlights where AI self-improvement is deeply hard and where practical solutions can work.
What this means in practice
- •For ai system developers: Design self-improving AI agents with bounded evaluation to ensure updates can be reliably assessed within resource limits.
- •For software verification engineers: Understand inherent limits when automating correctness checks for software updates proposed by autonomous agents in unrestricted environments.
A theory result. No direct application yet.
Authors
Brian B. Moser, Stanislav Frolov, Tobias C. Nauen, Federico Raue, Andreas Dengel
Abstract
With the rise of large language models, agentic systems are becoming increasingly autonomous, as they write code, call tools, revise prompts, and propose modifications to their own components. This autonomy raises a fundamental question for safe self-improvement: can an automated gate like a reward model or an automated reviewer reliably decide whether a proposed update should be accepted? The difficulty is structural. The gate must be a terminating, computable procedure, while the property we ultimately care about, namely whether an update is a genuine semantic improvement, depends on potentially unbounded program behavior. We formalize this mismatch by modeling agents as programs and defining improvement in terms of a semantic score that need not be computable. We study two evaluation abstractions. The first is a certifier that approves or rejects an update. The second is a search procedure that, given access to a proposal generator, attempts to output some improving candidate if one exists. We show that, in the fully general setting, no terminating procedure can simultaneously guarantee reliability (never accepting a non-improvement) and non-conservatism (never missing an available improvement): self-referential constructions can force any such gate or searcher to fail. Randomized gates with bounded error face the same limit. We then identify a standard decidable regime used in practice (bounded evaluation and a finite proposal budget) under which exhaustive simulation yields a terminating search procedure with the desired guarantees. We also show the limit of this recovery: a chain of accepted updates can raise the bounded score in every round while the true score falls. Our results draw a boundary between unbounded, universal settings where correctness guarantees collapse and bounded settings where they can be recovered by design.