Energy cost of large language model tokens grows nonlinearly
The SIMPLE Law for Energy Cost of LLM Inference
Artificial IntelligenceMachine LearningPerformance
Summary
Charging for generative AI by tokens ignores that not all tokens cost the same amount of energy. The authors found that each token increases memory usage and computational cost for the following tokens, making energy use rise faster than expected. They created a simple mathematical model called SIMPLE that accurately predicts how much energy different parts of the AI process use without needing to adjust it for each AI model. Their findings help explain why longer inputs and larger models use energy much faster, and how some techniques can save energy during inference.
What this means in practice
- •For ai infrastructure teams: Estimate and optimize energy consumption of large language model inference across different input lengths and model sizes with a general predictive model.
- •For cloud service operators: Better plan and bill generative AI services by understanding nonlinear token energy costs and impacts of caching or agentic serving approaches.
Authors
Hiari Pizzini Cavagna, Andrea Proia, Daniele Cesarini, Zeynep Kiziltan, Andrea Bartolini
Abstract
Generative AI is billed per token, yet tokens are not equally expensive. Each token enters the KV-cache and raises the cost of those that follow, so energy grows non-linearly along a sequence whose tokens look identical. No analytical model states what a token costs, and the most accurate SOTA model is a fit, not a law: it absorbs architecture into fitted coefficients, requiring per-LLM fitting. We derive closed-form expressions of the computational and memory-access complexity of LLM inference in terms of architectural hyperparameters, covering MHA and GQA, and compare a family of analytical models of increasing granularity. From these, we obtain the SIMPLE Inference Model to Predict LLMs Energy (SIMPLE), a four-parameter law that separates prefill from decode, and decode into KV-cache cost and next-token computation. Over traces of 13 dense LLMs from six families on NVIDIA H100 GPUs, it attains R2 = 0.969 and 11.97% MAPE, 3.1-3.5x below the SOTA, without per-LLM calibration. The SIMPLE law explains as well as predicts: prefill energy grows quadratically with the hidden dimension, so larger LLMs suffer most from long inputs; and more aggressive GQA KV grouping reduces the per-token energy by up to 3x relative to standard MHA. Increasing the reasoning token budget pushes output length beyond the analytically derived energy minimum, driving super-linear energy growth; although accuracy grows with the budget, energy grows faster, so Intelligence per Joule peaks at an intermediate budget and falls by up to 3.14x across a 4.0x reasoning-budget range. Agentic serving raises energy per request by 13.0x relative to non-agentic ones, driven by the growth of both the prefill and the KV-cache terms with input length; while prefix caching removes most of the prefill cost (-93%), the KV-cache term is unaffected, so even a cached agentic request costs 4.8x more.