Energy efficient cooling and gpu control cut ai datacenter costs
ETCInfer: An Energy-efficient Thermal-aware Cooling-joint Scheduler for LLM Inference in AI Datacenters
Distributed, Parallel, and Cluster ComputingPerformance
Summary
Running powerful language models in datacenters causes big energy use and heat. The authors study how to control cooling and GPU speed together to save energy without slowing down tasks or overheating. They built a system called ETCInfer that predicts heat and adjusts cooling, GPU speed, and batch size to keep everything safe and quick. Tests show ETCInfer cuts energy use by up to a third and greatly reduces overheating risks, while keeping delays very low.
What this means in practice
- •For datacenter operators: Reduce energy consumption and prevent GPU overheating in AI datacenters by jointly controlling cooling and GPU settings during LLM inference.
- •For cloud infrastructure teams: Improve energy efficiency and latency reliability for AI inference workloads by dynamically adjusting environmental and hardware parameters.
Authors
Rui Lu, Rui Ge, Huanghuang Liang, Xiaobo Zhou, Dan Wang
Abstract
Large language model (LLM) inference in AI datacenters creates a coupled control problem between GPU serving and facility cooling. Raising ambient temperature setpoints can reduce cooling energy and carbon, but also shrinks thermal headroom, induces GPU throttling, and leads to Service-Level-Objective (SLO) violations. In this paper, we study joint cooling--computing control for LLM inference: minimizing per-job GPU-plus-cooling energy while satisfying thermal safety and latency SLO constraints. We present ETCInfer, an energy-efficient, thermal-aware scheduler that selects a pre-job Computer Room Air Conditioner (CRAC) setpoint and adapts per-GPU frequency and micro-batch size during execution. ETCInfer builds compact physics-informed control models by calibrating GPU heat generation, chassis heat dissipation, CRAC power, and prefill/decode latency relations from telemetry. These models estimate hidden thermal states and time-to-throttle, enabling the scheduler to evaluate energy, temperature, and latency before applying an action. We formulate this joint setpoint--frequency--micro-batch control problem as a partially observable Markov decision process and design ETCAdapter, a learning-based controller that minimizes per-job energy under thermal safety and SLO constraints. We implement ETCInfer as a coordination layer over typical inference and cluster management stacks. Evaluation across real-trace simulation and validation experiments shows that ETCInfer reduces total job energy by up to 33.1%, thermal throttle exposure by up to 92.9%, and keeps SLO violation rates below 0.7% even at ambient temperatures up to $48^{\circ}\mathrm{C}$.