Agent reliability improves sharply with typed network interfaces
ANI-Gamut: Benchmarking Agent Reliability across the Gamut of Agent-Network Interface Abstractions
Networking and Internet Architecture
Summary
This paper studies how language-model software agents that manage computer networks perform depending on how they interact with the network. The authors test different ways agents can communicate, from raw commands to structured, typed instructions. They find that agents using structured, typed interfaces make far fewer errors and work more efficiently than those using raw command lines. The network setup also protects unrelated services when errors happen, so damage is prevented regardless of the agent style. This helps clarify the best way to build reliable network-managing agents.
What this means in practice
- •For network operators: Choose structured typed interfaces for automated agents to reduce errors and operational costs during live network changes.
- •For cloud infrastructure teams: Use tenant-isolated substrates to confine failures of network automation agents, preventing impact on coexisting services.
Authors
Lorenzo Bracciale, Pierpaolo Loreti, Andrea Mayer, Stefano Salsano, Wim Henderickx
Abstract
Large Language Model (LLM) agents are increasingly trusted to operate live networks: they read state, change configuration, and verify the result. A first-order question is left implicit: at which level of abstraction should the agent operate? We make the interface-abstraction level an explicit, controlled experimental variable, organizing agent-network interfaces into a spectrum from raw CLI (A0) through bounded wrappers (A1) and standardized model-driven configuration (A2) to typed transactional service intent (A3-T), reconciled source-of-truth automation (A3-R), and their combination (A4). We present ANI-Gamut, a reproducible, open-source playground that exposes the same task at several levels on a single, densely populated brownfield substrate, where many coexisting services share resources and collateral damage actually arises. We instantiate and measure four points of the spectrum and describe the others only at the conceptual level, and record three dependent variables as the substrate is stressed by injected faults: task reliability, collateral damage against pre-existing tenants, and operational cost. In a pilot with small run counts (n=20, n=6 and n=4 per level), the interface level moves reliability and cost sharply: on a live change-set task a raw-shell agent fails on all six seeds while a typed transactional interface succeeds on all six (paired McNemar p=0.031, on six discordant pairs), at roughly an order of magnitude less cost, as the engineering effort migrates from the agent to a reusable transaction layer. Collateral damage, by contrast, is absent at every level, whether benign or under faults: in a tenant-isolated substrate the agents fail safe, and the blast radius is held by the substrate's isolation, which moves the safety question from the agent to the substrate.