Large language models assessed on engineering their own infrastructure
$Φ$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?
Computation and Language
Summary
Building and improving the complex systems that run large language models (LLMs) is a big challenge. The authors present Φ-Bench, a new test designed to check how well LLMs can handle tasks like writing code for LLM infrastructure and optimizing entire systems over time. Their experiments reveal what current models can do and where they fall short in managing these complicated jobs. This helps understand how close we are to letting AI improve the tools that support it.
What this means in practice
- •For ai infrastructure teams: Evaluate and improve AI infrastructure engineering by measuring model capabilities on diverse engineering tasks across the LLM stack.
- •For software developers: Assist in debugging and completing complex code for machine learning systems by using LLMs tested on tasks spanning from small functions to full system design.
Authors
Leilei Ding, Shumin Wang, Yuting Huang, Fanqi Wan, Yinmin Zhang, Qi Han, Yiming Xu, Feiyuan Zhang, Xiaomeng Chu, Guoliang You, Wuyang Zhang, Daxin Jiang, Yanyong Zhang
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present $Φ$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack. Derived from optimization problems studied in frontier research and grounded in real-world code repositories, $Φ$-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure.