Power efficient mobile large language model inference improves speed and battery life
PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling
Machine LearningComputation and LanguageOperating SystemsPerformance
Summary
Running large language models on phones is hard because they use lots of power and create heat, which can slow down the phone. The paper presents PELM, a way to save energy by cleverly guessing parts of the model's output and adjusting processor speed based on how complicated the task is. This helps phones run these models faster and use less battery without losing quality. The authors tested PELM on different devices and found it can speed up processing by up to 23% and cut energy use by half.
What this means in practice
- •For mobile app developers: Integrate energy-saving strategies to run large language models efficiently on mobile devices, improving app responsiveness and battery life.
- •For embedded system engineers: Design on-device AI solutions that maintain performance while reducing heat and power use using multi-dimensional workload tuning.
Authors
Weisi Yang, Stephen Xia
Abstract
Deploying Large Language Models (LLMs) directly on mobile platforms at the edge is gaining traction due to a myriad of benefits, such as increased privacy, personalization, and reduced latency. However, LLMs have heavy computational requirements, which are difficult for resource-constrained mobile and edge platforms to fulfill. In addition to limited compute resources, mobile and edge systems often have a compact form factor and lack physical mechanisms to dissipate heat generated from high processor usage rates (e.g., fans) to prevent throttling and reduced processing power, which LLMs can easily cause. To mitigate these effects, prior works have proposed various power governing strategies, such as dynamic voltage and frequency scaling (DVFS), for reducing power and heat generation for heavy computational tasks on mobile platforms. Recently, DVFS methods tailored for mobile LLMs have also been proposed. However, these methods mostly focus on optimizing hardware parameters and processor frequencies, and they fall short under some thermally constrained scenarios. Drawing from recent advances in machine learning, we identify and take advantage of the key insight that not all tokens require full-depth inference to maintain high-quality generation. Motivated by this, we present PELM, a solution that augments traditional DVFS processor frequency tuning with two additional workload-specific knobs: 1) speculative decoding and 2) variable verification depth to expand the optimization space to multiple dimensions for more power efficient on-device LLM inference. In extensive evaluations across hardware platforms and datasets, PELM demonstrates superior performance compared to state-of-the-art power governing methods, with up to 23.1% speedup and 52.4% reduction in energy consumption, while maintaining comparable task performance. The source code is available at https://github.com/imec-nu/PELM.