Energy use of input tokens rises sharply in large language models
The Cost of Context: Profiling the Energy Footprint of Input Tokens in Large Language Models
Distributed, Parallel, and Cluster Computing
Summary
Processing long inputs in large language models (LLMs) uses more energy than previously thought, especially in the initial stage where the model reads all the input tokens at once. The authors studied two LLMs running on Nvidia hardware and found that the way attention works causes energy use to increase faster than earlier system studies suggested. They also showed that special memory methods can make this process more efficient, delaying big energy spikes by a lot. Finally, they compared the energy cost of reading inputs versus producing outputs, finding that optimized systems can narrow this gap but unoptimized ones keep it large and constant.
What this means in practice
- •For cloud platform operators: Optimize data center energy management by understanding and reducing the energy cost of processing long input contexts in LLM inference workloads.
- •For ai infrastructure engineers: Design more energy-efficient LLM serving systems by applying memory orchestration techniques like PagedAttention to delay costly computational scaling.
Authors
Boris Ruf, Marcin Detyniecki
Abstract
The rapid adoption of Retrieval-Augmented Generation (RAG) and long-context LLMs has shifted the computational burden toward the prefill phase---the initial stage where the model processes all input tokens in parallel before generating a response---yet its energy impact remains poorly characterised. We investigate the contradiction between massive hardware-level power spikes and the sub-linear marginal cost of input tokens reported in system-level studies. Through high-resolution telemetry of Llama-3.1 and Qwen-2 models on NVIDIA A100 hardware, we identify the efficiency threshold where $O(n^2)$ attention complexity overcomes GPU parallelisation. Our results quantify how optimised memory orchestration (PagedAttention) delays this scaling bottleneck across three orders of context scaling, yielding up to an 8.9$\times$ energy efficiency advantage at 112,000 tokens compared to standard baselines. Furthermore, we characterise the energy disparity between input and output tokens, demonstrating that optimised serving (vLLM) causes this ratio to narrow (from $\approx$850$\times$ to 750$\times$) as context scales due to high decode efficiency, whereas unoptimised baselines show a much smaller, roughly constant gap ($\approx$150--230$\times$) because their prefill is itself expensive. This work establishes context-scaling laws that differentiate engine-level optimisations from inherent model constraints, providing an empirical framework for sustainable LLM deployment.