Lightweight C code generator speeds up AI on tiny devices
A lightweight C Code Generator for Heterogeneous TinyML Systems
Software Engineering
Summary
Running AI models on very small devices with limited power and memory is tough because training and running these models often use incompatible tools and hardware. The authors designed a simple tool that generates efficient C code to run AI models on specialized small processors, like those used in radar systems. Their approach makes the AI run over twice as fast and uses less memory than a popular existing method. This makes it easier to deploy AI in tiny gadgets that do not have much computing power.
What this means in practice
- •For embedded systems engineers: Build faster and smaller AI applications on custom low-power hardware like radar processors using the presented code generator.
- •For iot device developers: Improve AI inference efficiency in resource-constrained edge devices by applying the tool’s optimizations when generating C code.
Authors
Chen Liu, Bernhard Vogginger, Christian Mayr
Abstract
Over the past decade, the significant advancements in artificial intelligence have been closely tied to increasingly sophisticated training frameworks and steadily powerful computing hardware. However, the entire ecosystem remains fragmented. On one hand, models are represented with frameworks dependent formats, while on the other hand, new tailored hardware for accelerating novel operators has continued to emerge. To effectively diminish the transfer costs from training to inference, we propose a lightweight C code generator designed for bare-metal heterogeneous systems that incorporate domain-specific accelerators. Through a comprehensive case study on a RISC-V based radar DSP, we demonstrate that by employing an accelerator with ahead-of-time optimizations like weight tuning and bias fusion, our approach achieves a 2.3 times improvement in inference speed and a 20% reduction in binary size compared to the microTVM flow registered with the same accelerator, thus offering another compelling choice for resource-constrained edge devices. The code is open-sourced in Gitlab.