Predicting inference cost speeds up diffusion large language model serving
Denoising Surface: Modeling and Predicting Inference Cost for Diffusion LLM Serving
Machine Learning
Summary
Diffusion large language models (dLLMs) are powerful but estimating how long they take to respond is tricky because their work happens in steps and blocks. The authors saw that common ways to guess the cost are too simple and miss important details. They invented a method called the Denoising Workload Surface (DWS) that captures this two-dimensional step-block pattern to predict inference cost more accurately. Their predictor runs fast on CPUs and adapts across different hardware setups. Using DWS improves cost predictions and reduces delays in real chatbot services.
What this means in practice
- •For cloud operations teams: Manage computational resources more efficiently by accurately predicting the inference cost per request in dLLM-based services.
- •For real-time chatbot developers: Reduce user-perceived delay in chatbots by scheduling requests based on precise per-request cost prediction from the DWS method.
Authors
Haoyu Zheng, Fangcheng Fu, Binhang Yuan, Yongqiang Zhang, Liang Deng, Hao Wang, Yuanyuan Zhu, Xiao Yan, Jiawei Jiang
Abstract
As diffusion large language models (dLLMs) become more capable, they are moving from research settings to real-world \textit{serving}, where request management (such as scheduling and resource allocation) relies on accurate estimation of per-request inference cost. However, common cost proxies fall short for dLLMs: output length ignores that one forward pass can unmask multiple tokens, and denoising-step count ignores the \textit{heterogeneous} per-step costs. We observe that the block-autoregressive generation mechanism induces a two-dimensional execution structure over output blocks and within-block denoising steps, whereas these proxies collapse it into a scalar, discarding information essential for characterizing the cost. Motivated by this insight, we propose the Denoising Workload Surface (DWS), which preserves this two-dimensional block-step structure as a probability surface to weight the heterogeneous per-step costs. We then design a coarse-to-fine training scheme that enables a lightweight prompt-only predictor to accurately predict the complex DWS. This predictor runs efficiently even on a single CPU core, avoiding GPU contention with the serving model. Since DWS decouples request-dependent execution behavior from deployment-specific cost factors, the predictor transfers across hardware configurations without retraining. In \textit{real-world} serving experiments, DWS reduces cost-prediction error by up to $2.50\times$ over scalar-based predictors, while the DWS-guided shortest-job-first scheduler reduces end-to-end latency by up to $1.92\times$ for online chatbots.