HeatCache schedules LLM inference to save energy and reduce overheating
HeatCache: Thermal-aware Energy-efficient LLM Inference Scheduling for Chassis-level Liquid Cooling in Sustainable Edge Server Rooms
Distributed, Parallel, and Cluster ComputingPerformance
Summary
Large language models (LLMs) running on multiple GPUs use a lot of electricity and generate heat, which can cause hardware issues if cooling is less efficient. The authors present HeatCache, a new scheduler that carefully plans how requests are run on GPUs cooled by liquid systems to save power and avoid overheating. HeatCache treats the cooling system as a temporary heat buffer to manage workloads without breaking performance rules. Their tests show significant energy savings and much less overheating while keeping service delays low, even at higher ambient temperatures.
What this means in practice
- •For edge server operators: Manage large-scale language model services on liquid-cooled GPU servers more efficiently with reduced energy costs and better thermal management.
- •For data center cooling teams: Coordinate GPU workload scheduling with chassis-level liquid cooling systems to improve server room sustainability at higher ambient temperatures.
Authors
Rui Lu, Huanghuang Liang, Kaiqi Guan, Dan Wang
Abstract
LLM inference is increasingly deployed at institution-scale edges to meet service requirements. However, multi-GPU inference consumes a large amount of electricity and produces substantial heat. To improve sustainability, operators and regulations often demand raising the ambient setpoint to reduce cooling electricity. This can increase thermal throttling and hardware aging, leading to Service-Level Objective violations. In this paper, we present HeatCache, a thermal-aware, energy-efficient LLM inference scheduler for commercial chassis-level AIO liquid-cooled GPUs at sustainable ambient temperatures. HeatCache treats AIO loops as a temporary heat buffer, measured by heat budget and schedules requests to minimize energy subject to thermal safety and SLO constraints, based on an electrical-informed heat-demand estimation from HeatiTS. We implement HeatCache atop vLLM and show that it reduces computing energy by up to 18.0%, decreases thermal-throttle exposure by 81.7%, and maintains SLO violation rates below 0.9% even up to $48~^{\circ}\mathrm{C}$.