Energy use of vision language models on raspberry pi and jetson devices
Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms
Machine Learning
Summary
Running vision-language models that combine pictures and text on small computers can use a lot of energy and create heat. The authors studied two small computers, a Raspberry Pi with a special chip and a Jetson device with a graphics chip, to see how much energy they use and where it goes. They found that using the special chip on the Raspberry Pi saves a lot of energy and keeps the computer cool. They also saw that making the computer answer with text costs more energy than looking at the image. Different devices work better depending on how long the text answers are.
What this means in practice
- •For embedded system engineers: Optimize energy usage and thermal performance when deploying vision-language models on small devices by choosing between NPU or GPU accelerators based on output length.
- •For mobile device developers: Design energy-efficient applications that run vision and language models locally by understanding fixed and per-token energy costs for image and text processing.
Authors
Junlin Liao, Harshit Kondle, Di Liu, Zonghua Gu
Abstract
Vision-language models (VLMs) are moving onto edge devices, but little is known about how much energy they draw on single-board computers with NPU or GPU accelerators, or where that energy goes. We measure total-system energy, latency, and thermal behavior of Qwen2-VL-2B-Instruct and its text-only sibling on a Raspberry Pi 5 with a Hailo-10H NPU and on a Jetson Orin Nano Super, on both CPU and accelerator, and separate the fixed energy of each request from the energy that grows with output length. Because the accelerators use different quantization and serving software, we compare deployment stacks rather than silicon. Three findings stand out. First, NPU offloading makes the Pi a much more efficient VLM platform: it cuts energy per generated token by about two-thirds, leaves the host CPU largely free, and avoids the thermal throttling that the CPU backend reaches in sustained use. Second, an image adds both fixed and per-token energy, but at typical response lengths generating the answer costs far more than processing the image. Third, the Jetson GPU favors short responses through its lower fixed cost per request, whereas the Pi NPU favors longer ones through its lower cost per output token; their energy curves cross near 40 output tokens.