Comparing vector and CGRA chips for AI nonlinear functions

To SIMD or Not to SIMD: Comparing RVV vs. CGRA SoCs for Non-Linear Functions in AI

Hardware ArchitectureArtificial Intelligence

Summary

Non-linear functions play a big role in AI models, but the common chips used to run them haven’t improved as fast as those for matrix math. The authors built and tested two types of computer chips to see which works better for these functions: one using a common vector processor and the other using a more flexible compute design called CGRA. They found that CGRA chips are quicker on simpler functions like SiLU, while vector processors perform better on complex functions like softmax. Overall, each chip type has strengths depending on the function complexity in AI tasks.

What this means in practice

  • For ai hardware engineers: Choose between vector processors and CGRA for efficient hardware design depending on whether AI tasks use simple or complex nonlinear functions.
  • For embedded system developers: Optimize edge AI devices by selecting appropriate SoC architectures for neural network activation functions based on function complexity.

Authors

Sameer Deshmukh, Naoya Mizuki, Takumi Honda

Abstract

Non-linear functions are an important class of operations in modern large language models. However, chip vendors building for AI have so far relied on traditional vector or scalar processing units for non-linear functions. The performance of these units has not kept pace with matrix multiplication units (MMUs), which can be found in both high-performance and edge-computing devices from almost every vendor. Reconfigurable units such as FPGAs and CGRAs have been proposed in the literature, but have not been systematically compared with traditional units backed by MMUs. In this paper, we build two SoCs: one using an RVV vector processor and the other using a latency-insensitive (\emph{elastic}) CGRA, both coupled with an MMU and driven by a single-core CPU. We then propose mappings for common feed-forward networks on our SoCs and find that the CGRA reaches a maximum speedup in clock cycles of 7.12x over RVV on the activation stage of simple point-wise non-linear functions like SiLU, whereas the RVV reaches a maximum speedup of \textbf{9.8x} over the CGRA on complex non-linear functions such as $\mathrm{softmax}$. Over the enclosing feed-forward block the corresponding maxima are 3.16x and 4.24x.