Self evolving inference system greatly speeds up language model serving
SEIS: Self-Evolving Inference Systems
Artificial Intelligence
Summary
Running large language models quickly and cheaply is important, but usually people only improve small parts of the system. The authors created SEIS, a system that improves itself step-by-step without human help to make the entire language model engine faster. This self-improving engine ran tests on a popular language model and was over three times faster than the original, even beating some top commercial engines. The authors confirmed that the system still produces correct results. The work shows self-evolving technology can boost complex software as a whole.
What this means in practice
- •For ai infrastructure teams: Optimize entire language model inference systems without manual tuning to gain large speedups on GPUs like the H100.
- •For cloud service providers: Increase throughput of hosted language model APIs by using self-evolving inference engines that improve over time autonomously.
Authors
Zhen Xu, Jingyu Liu, Zongze Li, Tahseen Rabbani, Ce Zhang
Abstract
Inference systems determine how fast and how cheaply language models can be served, so making them faster has direct practical value. However, prior work focuses mostly on optimizing certain parts such as kernels or memory within the large system. In this work, we take a holistic approach and apply agentic self-evolution to optimize the whole system end-to-end. Our SEIS (Self-Evolving Inference Systems) autonomously optimizes the entire mini-sglang engine without human intervention through iterative sessions with inherited experiences and code changes. Serving Qwen3-0.6B on H100, the resulting engine reaches 3.27X the throughput of the original mini-sglang implementation and beats SOTA engines like vLLM, TensorRT-LLM, and SGLang in the single-request workload. The correctness of the optimized inference engine by SEIS is tested in terms of numerical difference and downstream accuracy on math and long-context retrieval tasks. The code and session histories show that the speedup comes from redesigning the whole engine and that building on earlier sessions beats independent attempts. These results suggest that agentic self-evolution can optimize a complex system end-to-end. The evaluation also has to evolve with the engine, and letting agents evolve it is a natural next step.