Summary
This paper looks at when an AI system's defenses can become strong enough to stop attacks by constantly finding and fixing problems. The authors show that if every attack has a chance to be found, fixes work well, and updates keep earlier protections, then the system will eventually defend all possible attacks for a fixed set of inputs. They also study how this defense works when new attacks appear or repairs cover groups of attacks. Additionally, they model a game between attackers and defenders to find the best way to spend resources to prevent attacks. Their experiments show that faster fixing helps shorten attack damage even if it does not lower the chance of getting attacked.
What this means in practice
- •For ai security teams: Plan investments in automated testing and repair methods to achieve guaranteed protection against attacks on a fixed input set.
- •For cybersecurity operations: Optimize resource allocation between attack discovery and repair processes to deter attackers cost-effectively in real-world threat scenarios.
A theory result. No direct application yet.
Abstract
Feedback from automated testing, human red teaming, and incident response can strengthen an AI system's defenses when discovered failures lead to effective repairs. We study when this feedback process provides sufficient protection and when investing in it is economically worthwhile. We begin by showing that an attack surface composed of finite number of inputs is defended with probability 1 if every unresolved attack has a persistent chance of discovery, repairs are effective, and subsequent updates preserve earlier protection. We derive completion-time bounds and extend the analysis to growing attack surfaces, repairs that generalize across related attacks, and multiple discovery mechanisms. These results distinguish eventual protection against each fixed attack from complete protection at a single time. We then formulate a defender-led Stackelberg game in which the defender invests in proactive discovery and reactive repair, anticipating the attacker's choice of search effort. We characterize the least-cost allocation that deters attack and the equilibrium regimes in which the defender funds neither capability, one capability, or both. Numerical experiments illustrate these regimes and show how faster repair can reduce compromise duration without reducing compromise probability.unified theory of performance limits in generative language models.