LLM performance gaps exposed in network control with black-box methods
Blackbox Performance Analysis of LLM-based Network Control
Networking and Internet ArchitectureMachine Learning
Summary
Networks rely on complex computer models to manage traffic and resources, but new AI language models (LLMs) are making these tasks smarter. The authors created LENS, a tool to test how well these AI models work when given new and tricky tasks, even though the AI's inner workings are hard to understand. LENS uses smart search methods to find where the AI struggles, helping improve network control systems that manage things like internet traffic or GPU usage. This approach finds big problems the AI has and performs as well as other tools that need full access to the AI's code.
What this means in practice
- •For network schedulers: Detect critical performance weaknesses in AI-driven traffic engineering to improve network reliability before deployment.
- •For datacenter operators: Test AI-based GPU allocation systems to find scenarios where resource management might fail or underperform.
Authors
Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel, Ramesh Govindan
Abstract
LLMs are becoming an indispensable part of network control systems due to their impressive reasoning. Before such LLM-enabled systems can be deployed, it is imperative to develop tools that can audit their performance on previously unseen inputs. However, auditing these systems is challenging due to the inherent opacity of LLMs. LENS is the first step towards black-box performance analysis of LLM-enabled network control systems. It uses novel methods to extend deep reinforcement learning to guide the search for performance gaps and inputs that make LLMs struggle. LENS finds large gaps for LLM-based traffic engineering and GPU allocation systems, and even matches existing white-box analyzers designed for hand-crafted TE heuristics.