Summary
The paper looks at how artificial intelligence agents, like large language models, are becoming central to computer systems and can perform important actions like editing code or managing emails. These AI agents have control similar to the core of an operating system but lack traditional security checks. The authors identify key security challenges, especially around verifying what is safe input and what actions are authorized. They categorize defenses and highlight that many current security tests might overstate safety. Finally, they explore future designs where AI models themselves manage security decisions directly.
What this means in practice
- •For operating system developers: Design new operating systems that integrate AI agents securely by applying the proposed architecture and mediation framework.
- •For cloud infrastructure teams: Improve security of AI-driven services by understanding where mediation gaps occur and how to reduce undetected attacks.
A position paper. It proposes an approach and reports no results.
Abstract
Large language model agents are now privileged principals that take consequential actions: editing code repositories, operating inboxes, completing purchases. Their authority is kernel-grade, but it comes without what classical systems security requires: a trusted mediator interposed on every access. Operating-system vendors are now rebuilding the platform around this de-facto agent kernel, inheriting complete mediation as a design problem. We systematize the security of such systems around a single distinction: a crossing mediated over provenance admits a deterministic check, while one over content semantics does not. A trust-boundary taxonomy locates where mediation must occur and isolates the central mediation gap at two kinds of semantic judgment: distinguishing data from instruction in untrusted input, and an authorized action from an unauthorized one. We argue that this gap leaves an irreducible residual of undetected attacks wherever inputs and actions are not restricted in advance to an enumerated set. The same distinction makes attack-success statistics actionable, placing each number on a spectrum from deployment debt (a sound deterministic mediator left unused) to a structural gap (no such mediator known). We systematize defenses across runtime monitoring, architectural separation, and authorization, and show that current evaluations tend to overstate deployed security through evaluation-validity failures. Finally, we carry that analysis forward beyond the de-facto kernel, to an architecture in which the model itself becomes the arbitration core, and derive the design constraints, open challenges, and research agenda for a security-first AI-native OS.