Datacenter GPUs save power with model-specific phase-adjusted control
Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
Running large AI language models in datacenters uses a lot of GPU power, which limits how many requests they can handle. The authors found that using a fixed power setting for all parts of the AI task wastes energy or slows things down. They designed a new system that sets power limits separately for each task phase, calibrated to the AI model and latency needs. This method saved up to 20% energy per output token while keeping response times fast, especially for certain model types.
What this means in practice
- •For datacenter operators: Implement model-specific, phase-separated GPU power control to improve energy efficiency in AI inference clusters without compromising request latency.
- •For cloud ai service providers: Optimize large language model serving infrastructure by calibrating power caps per task phase to reduce operational power costs under real workloads.
Authors
Jae Gon Kim, Donghoon Yoo, Hanyul Ryu, Sungho Ha, Juyeon Lee, Soojung Ryu
Abstract
Datacenter GPU power is the binding constraint on LLM serving capacity, and production serving has shifted to prefill/decode (PD) disaggregation. Deploying NVIDIA's Max-Q inference profile on a disaggregated B200 system, we found its realized gain modest (+8.6% tokens/J), model-dependent, and carrying a mean end-to-end latency cost (+5.2%) that throughput-only evaluation does not surface; the profile also applies one setting to prefill and decode GPUs that operate in opposite hardware regimes. We hypothesize that the optimal power setting is a property of the deployed (model, quantization, engine, hardware) combination rather than of the GPU class, that each lane warrants its own profile, and that converting SLO headroom into energy safely requires latency-gated calibration under a runtime SLO guard rather than a fixed recipe. We present a phase-decoupled, model-calibrated controller: the prefill lane runs under an SM-clock window whose floor is a latency guarantee by construction, and the decode lane under a power cap placed by automatic calibration just above a measured throughput/latency cliff. Because a disaggregated decode lane draws flat, memory-bound power, the cap binds continuously, the reactive-overshoot weakness that led POLCA to reject capping is absent, and the GPU's own power manager retains throughput under the cap. On an 8x B200 node serving Qwen3-Coder-480B (FP8) under agentic load, our balanced mode delivers +20.4% tokens/J at +3.5% mean e2e versus +8.6% at +5.2% for Max-Q, a Pareto improvement on both axes. On Qwen3-235B-A22B (NVFP4) every operating mode meets the ITL-p99 SLO in every repetition; both vendor profiles miss it. A decode-actuator A/B shows the calibrated cap beats static clock locks, and a three-day sustained run saves 32.3% of a lane pair's electricity. Both models are MoE; a dense model recovers roughly 5x less, so we scope our claims to MoE serving.