Framework speeds up transformer models on edge devices with risc v
A Framework for Accelerating Transformer Inference on RISC-V for Edge AI
Hardware Architecture
Summary
Running large language models like transformers on small devices such as IoT gadgets is challenging because these devices have limited computing power. The authors created a special framework that adds new instructions to a RISC-V processor to make important calculations faster for these models. They also used a clever technique called loop unrolling in software to speed things up without extra hardware cost. Their tests showed almost twice as fast inference times with only modest increases in chip size and power use.
What this means in practice
- •For embedded system developers: Build faster natural language processing on resource-limited IoT devices by using the proposed RISC-V extensions and instruction optimizations.
- •For chip designers: Integrate custom instructions into RISC-V cores to enhance transformer inference speeds with limited area and power overheads for edge AI chips.
Authors
Ajay Kumar M, Vishnu PS, Yike Li, Shreejith Shanker, Dimitrios S. Nikolopoulos, Bo Ji, Hans Vandierendonck, Deepu John
Abstract
This work presents a framework for accelerating transformer-based language models (LMs) on resource-constrained IoT devices. The framework targets compact LMs: BERT-Tiny (B-Ty), MobileBERT (M-Bt), MiniLM (M-Lm), Electra (E-Lt) and DeBERTa (D-Bt) -- selected for their architectural diversity and use in edge inference scenarios. The proposed flow derives lightweight instruction set extensions tailored to the non-obvious computational patterns of these models. In addition, a custom instruction is introduced to accelerate the address generation stage of batch matrix multiplication, achieving a performance improvement of 15.39--21.74% with modest ASIC overheads of 6.79% in area and 2.33% in power. To further enhance performance without incurring additional processor core hardware cost, an optional compiler-directed loop unrolling strategy is employed, trading increased code size for overall reduced execution time. Evaluation on the Synopsys trv32p3f RISC-V core demonstrates inference speedups of up to 2.19x, and FPGA implementation on the AMD Zynq UltraScale+ ZCU102 shows a 32.94% area overhead at 75 MHz, whereas the ASIC implementation using the TSMC 28 nm library incurs a 21.08% area overhead while operating at 250 MHz.