EPACT cuts cost and keeps deadlines in large AI model serving
ePACT: Energy-Performance-Aware Commitment Tracking for LLM Serving
Distributed, Parallel, and Cluster Computing
Summary
Running large AI models can be expensive because energy use may differ from set hourly commitments, causing extra costs. The authors designed ePACT, a system to carefully manage energy use and performance so these extra costs are minimized while still serving user requests on time. ePACT adjusts computing power and GPU speed in real time, predicting how different settings affect energy use and task completion. Tests show ePACT greatly reduces deviation costs compared to a prior method while still meeting service goals.
What this means in practice
- •For data center operators: Manage hourly electricity purchase commitments to reduce penalty costs while keeping AI service deadlines with real-time server adjustments.
- •For cloud platform engineers: Implement dynamic GPU performance controls that balance energy consumption against service speed to improve cost efficiency of large AI model hosting.
Authors
You Peng, Youhe Jiang, Chen Wang, Binhang Yuan
Abstract
Reducing LLM serving energy does not by itself guarantee lower deployment cost when electricity procurement exposes operators to unfavorable deviations from preset commitments. We study hourly commitments with positive, potentially asymmetric costs for overuse and underuse, and formulate energy-Performance-Aware Commitment Tracking: minimize deviation costs subject to request-level service requirements. We implement ePACT, a two-level controller that adjusts serving capacity and GPU clocks as requests arrive. A global planner updates interval energy targets from measured consumption and the remaining hourly commitment. A local decision maker predicts candidate configurations' energy and completion times, checks predicted deadline misses, and selects among admitted configurations by asymmetric target-deviation cost, with a service-first fallback. Coarse-to-fine action search runs asynchronously with serving. We evaluate ePACT through single-hour comparisons, controller ablations, and full-day trace simulations for H20 and H200 GPU pools. In the 24-hour simulations, ePACT reduces the asymmetric deviation cost by $73.8\%$ and $75.7\%$ relative to vLLM while retaining near-vLLM SLO attainment. Mean absolute hourly deviations are $2.16\%$ and $2.31\%$, respectively.