Large language models tested on network device routing setups
An Experimental Study on the Functional Correctness of Network Device Configurations Produced by Large Language Models
Networking and Internet Architecture
Summary
Setting up network devices correctly is important but can be hard. This paper looks at how well large language models (LLMs) do when asked to create routing setups for network devices without any extra help. The researchers checked the accuracy of these setups by comparing the resulting routing tables to the correct ones. They found that while LLMs can generate some valid configurations, there are clear limits to their current reliability for real-world network management. This work helps understand how useful LLMs might be for network administrators in practical situations.
What this means in practice
- •For network administrators: Use large language models as an initial check tool to generate basic routing configurations for Cisco-based devices without extensive manual scripting.
- •For it operations teams: Evaluate the readiness of language models to assist in automating network setup validation by comparing actual routing tables against expected ones.
Tested on one dataset.
Authors
Anastasios C. Politis
Abstract
This paper presents a preliminary study on the efficacy of network device configuration produced by Large Language Models (LLMs). Currently, the study focuses on routing protocol configurations in a predefined small internetwork with Cisco-based routing devices. The capability of three freely available LLMs with standard capabilities (i.e., with no enhanced features) to produce effective setups is investigated. Prompt engineering methods are excluded in order to assess the "first-response" configurations of the models under consideration. The assessment method is based on the contents of the resulting routing table, aiming at measuring the device functional correctness. For that purpose, the Jaccard index is used to indicate the percentage of correct network prefixes contained in the routing table constructed by the model-based configuration commands. The work provides insights on the maturity and limitations of modern LLMs on aiding network administrators with network device configurations in real deployments.