Agent safety persists through self-improvement despite challenges
Safety Must Survive Self-Improvement: Why Failures Persist and How Agents Recover
Software EngineeringCryptography and Security
Summary
When intelligent agents improve themselves step-by-step, keeping them safe is tricky because mistakes can persist over these improvements. The authors study how these safe or unsafe behaviors stick around and how agents can recover when they fail. They find that old mistakes can remain even when better fixes exist, but retesting and choosing carefully what to fix next helps agents recover safety efficiently. Their work shows it’s important to check what will actually run now and pick the right part to improve to keep safety intact.
What this means in practice
- •For autonomous system developers: Improve safety monitoring and recovery in self-modifying software agents by using score refreshing and targeted code edits.
- •For security software teams: Enhance security update processes for stateful authorization software to reduce persistence of unsafe versions after patches.
Authors
Yunbei Zhang, Janet Wang, Saiyue Lyu, Yingqiang Ge, Kaiqu Liang, Zijian Jin, Chandan K Reddy, Jihun Hamm
Abstract
Recursive self-improvement (RSI) allows agents to carry useful changes across generations. Maintaining safety across these generations involves both preventing unsafe behavior from persisting and enabling recovery when failures occur. We study these challenges through a controlled testbed of stateful authorization tasks, where fixed LLM editors optimize executable agent components and independent traces record their effects. Paired interventions separate which revisions pass validation, which program continues running, and which program the editor revises next. After a new authorization dependency invalidates previously tested optimizations, historical scores preserve the same unsafe programs in 22 of 48 framework histories despite a correct alternative in every affected archive. Refreshing scores restores correctness on the original suite, with residual failures on independently composed tests. Failures also persist under an unchanged contract when all proposals are rejected and the failed incumbent remains active. Starting from shared failures, editing the initial implementation instead of the failed one improves recovery, although the advantage varies across editors. Full validation and validated rollback end with fully correct programs in the core trajectory study while retaining over 43% deployment savings. Preserving agent safety requires checking what will run under current conditions and choosing which implementation to edit next.