Edge ai chips differ widely in energy use and speed limits
TinyML Hits the Memory Wall: A Benchmark of Ten Sub-Watt Edge Devices
Hardware Architecture
Summary
Running smart AI programs on tiny devices that use very little power is tricky because not all chips are equally good at it. The authors tested ten small computers to see how much energy they use and how fast they are at working with popular AI models. They found that chips with special AI parts called NPUs are more limited by how fast they can move data in and out of memory, rather than just the number of calculations they do. This means counting calculations isn’t enough to judge how well these chips perform. Some newer chips use much less energy than others, helping choose the best for tiny AI tasks.
What this means in practice
- •For embedded systems engineers: Select low-energy microcontrollers with NPUs optimized around memory transfer to run AI models efficiently on battery-powered devices.
- •For wearable device developers: Choose edge AI hardware that minimizes energy use over idle to extend wearable device battery life while running complex neural networks.
Authors
Pietro Bonazzi, Luigi Capogrosso, Ahmet Celik, Rosanna Manzo, Marco Giordano, Szymon Ruciński, Jonathan Hungerbuehler, Michele Magno
Abstract
Neural network inference at the edge must fit tight energy budgets. Existing benchmarks compare few devices, do not report energy on a common basis, and omit the newest microcontrollers with an NPU or the 1 GHz Cortex-M85. Hardware-aware design also ranks networks by MAC count, yet no study has tested whether this proxy holds across accelerators. We benchmark sub-watt edge AI hardware by evaluating four CNNs (MobileNetV3, MCUNetV1, EfficientNet, SqueezeNet) and two tiny transformers (MobileViT-XXS, DeiT-tiny) on ten commercial devices: an in-sensor DSP (Sony IMX500), four MCUs with an NPU (STM32N6, Alif Ensemble E7/E8, GreenWaves GAP9), and five MCUs without an NPU (STM32U5, Apollo510B, ESP32-S3 Xtensa LX7, RA8M2, STM32V8). Four devices are novel (Alif E8, Apollo510B, RA8M2, STM32V8). Measured as energy above idle, the IMX500 (0.13-0.35 mJ) uses 1.3-3x less energy than the STM32N6 (0.31-1.05 mJ) on every model; the GAP9 uses the least on MobileNetV3 and the Alif E7 the least on MCUNetV1 and SqueezeNet. All four use 6-34x less energy than the STM32V8 (1.85-3.98 mJ), and the Alif E7 and IMX500 form the power-performance Pareto front. Latency on all four accelerators scales with memory traffic (bytes of weights and activations, r >= 0.98) and not with MACs, so they are memory-bound, while no MCU without an NPU is. The NPU advantage grows with intensity from 1.4x over a 1 GHz Cortex-M85 on MCUNetV1 to 8.2x on SqueezeNet. MAC count is therefore a misleading cost proxy for sub-watt accelerators.