RoofLang enables AI to design faster large language model inference systems
RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
Optimizing how large language models (LLMs) generate answers is tricky because current AI improvements rely on measuring existing software, which limits new ideas. To fix this, the authors created RoofLang, a specialized programming language that describes computing tasks in a way that allows AI to explore entirely new system designs. Using RoofLang, they found specific LLM designs that can be 3.5 to almost 40 times faster than others, mainly by improving memory use. The system also automatically discovered better designs that further boosted speed and responsiveness on real hardware.
What this means in practice
- •For ai infrastructure engineers: Design and implement more efficient LLM inference systems by using RoofLang to explore novel architectures beyond current software limitations.
- •For cloud service providers: Improve throughput and responsiveness of hosted LLM services by adopting RoofLang-enabled system designs optimized for hardware like NVIDIA B300.$Commercial implications: Enables selling faster and more responsive AI inference services to enterprises using advanced LLM architectures found by RoofLang.
Authors
Ziyue Yang, Yuting Jiang, Lei Qu, Peng Cheng
Abstract
AI is beginning to make substantive contributions to LLM inference optimization. Existing AI optimizations are predominantly profiling-based. Profiling-bound feedback confines the search to the capabilities and performance of an existing software stack, preventing a fundamentally better architecture of LLM inference systems from being identified. To enable the AI-driven LLM inference system architecting loop, we argue that a general workload representation, a verifiable mutation space, and an implementation-independent evaluator are required. We present the RoofLang domain-specific language (DSL) that provides these features. In our evaluation, RoofLang reveals that DeepSeek V4-series models could achieve 3.5-39.5$\times$ higher peak decode throughput than other representative models. This gap is disproportionate to their total parameter counts and arises largely from compact KV-cache designs that support larger batches and reduce memory traffic. A persistent optimizer agent further discovered several new architectures that improved both throughput and interactivity of DeepSeek V4 Pro on NVIDIA B300 by 6.23-50.1%.