AI agents struggle to fully fix errors in network experiment records
Can AI Agents Detect and Repair Artifact Drift in Network Experiments?
Networking and Internet ArchitectureArtificial Intelligence
Summary
AI agents can help detect and fix mistakes in records of network system experiments, but they often miss complex problems that involve understanding hidden relationships. The researchers created a test called NetArtifactBench to see how well these agents repair inconsistent records while keeping true statements intact. They found that agents do well with simple fixes but often fail when repairs require spreading changes across different parts of the record. This shows it’s important to design AI tools that not only finish tasks but also keep experimental records trustworthy.
What this means in practice
- •For network operators: Improve trust in network experiment data by automating the detection and repair of simple errors while preserving valid information in records.
- •For software developers: Build AI tools that maintain integrity of experimental logs by identifying complex inconsistencies requiring multi-artifact adjustments during debugging.
Authors
Tianzhu Zhang, Weichen Tao, Changgang Zheng, Yusheng Zheng, Long Chen, Xiaoyi Fan, Meikang Qiu
Abstract
In recent years, AI agents have evolved into capable assistants that carry out multi-step tasks in digital environments. The network systems community is beginning to explore these capabilities in operational and experimental settings. However, an agent operating in network systems should not be judged solely by whether it completes the immediate task. The experiment record it modifies must also remain trustworthy. We call this property artifact integrity: the record's claims must remain supported by the available evidence, confined to the scope established by that evidence, and traceable through the artifacts that encode their support. To make this property measurable, we introduce NetArtifactBench, which tests whether AI agents can repair inconsistent records derived from public network-system artifacts while preserving claims that remain supported. The benchmark contains 52 instances with injected inconsistencies ranging from direct contradictions to unstated relations spread across several artifacts. We evaluate 23 agent configurations across three general-purpose AI agent runtimes using deterministic scoring. The average contract pass rate is 65.3 % across 5,980 outputs, but no agent runtime exceeds 30 % when repair requires recovering implicit relations and propagating changes across artifacts. These results reveal a sharp boundary between local correction and complete record-level repair. Therefore, we argue that artifact integrity should become a first-class design and evaluation requirement for AI agents operating on network systems.