NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration
2026-08-24 • Networking and Internet Architecture
Networking and Internet ArchitectureArtificial Intelligence
AI summaryⓘ
The authors created NetConfArena, a new way to test how well large language models (LLMs) can handle configuring computer networks in realistic, risk-free settings. Unlike earlier tests, their system lets these models interact with multiple devices in a simulated network and checks if the network works correctly afterward. By running many tasks focused on different network protocols, the authors found that LLMs fail not just by making command mistakes but also by struggling to follow task rules and plan properly. Their results suggest improving these models by learning from successful runs and building better tools to monitor and guide their actions.
Large Language ModelsNetwork ConfigurationBenchmarkEmulationMulti-device NetworksProtocol ComplexityTopologyTask TemplateExecution TrajectoriesRobust Planning
Authors
Chang Liu, Xiaohui Xie, Xinyi Chen, Yong Cui
Abstract
Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.