AscSpMM speeds up sparse matrix math on Ascend neural processors
AscSpMM: Efficient SpMM on Ascend NPUs via Pipelined Vector-Cube Collaboration
Hardware Architecture
Summary
Multiplying sparse and dense matrices is important for many computing tasks but is hard to do fast on specialized neural chips because of how they handle data. The authors designed AscSpMM, a method that splits the work between two types of processors inside the chip, letting them handle irregular and regular parts separately. This speeds up computation a lot, reduces memory use, and is faster than existing methods on both Ascend and GPU hardware. It also uses smart buffering and tiling strategies to keep both processors busy.
What this means in practice
- •For graph analytics engineers: Accelerate large-scale sparse graph computations on Ascend NPUs using optimized SpMM to reduce runtime and memory footprint.
- •For scientific computing teams: Improve the speed and resource efficiency of sparse neural network training and simulations by employing AscSpMM on compatible NPUs.
Authors
Fei Li, Haoxin Li, Kehong Liu, Yanzi Zhou, Weiguo Wu, Shiqiang Nie, Jinyu Wang, Song Liu
Abstract
Neural processing units (NPUs) provide high compute density and energy efficiency through specialized matrix engines and software-managed on-chip memory. However, their libraries primarily target dense computation. Sparse matrix-dense matrix multiplication (SpMM), a key kernel in graph analytics, sparse neural networks, and scientific computing, requires index-driven operand selection, indirect addressing, scattered memory accesses, and load balancing. These requirements conflict with the fixed shapes and contiguous layouts preferred by NPU matrix engines. GPU Tensor Core SpMM methods cannot be directly transferred to NPUs with decoupled vector and matrix engines. We present AscSpMM, which separates irregular operand preparation from regular matrix computation through a parameterized Vector-CSR representation. The vector engine decodes indices and gathers and reorganizes dense operands, while the Cube engine performs matrix multiply-accumulate operations. A double-slot ring buffer enables dependency-aware asynchronous pipelining, while operand reuse and runtime-adaptive tiling reduce cross-engine stalls. On an Ascend 910B3, AscSpMM achieves geometric-mean speedups of 2.36x over GEMM, 19.92x over an existing SpMM implementation, and 5.82x over cuSPARSE on an A100. It reduces matrix-input storage and peak HBM footprint by 48.24% and 22.98% on average, while remaining competitive with specialized GPU Tensor Core SpMM kernels. The code is available at https://github.com/LfLab-AI/Ascend-NPU-SpMM.