Fpga design speeds up sparse matrix multiplication in longformer attention
A Pipelined FPGA Architecture for Banded Sparse Matrix Dense Matrix Multiplication in Longformer
Hardware Architecture
Summary
Longformer is a version of a popular AI model that pays attention to only a small, nearby part of a long input instead of everything, which saves computing power. This creates a special kind of pattern in the data called a banded sparse matrix. The authors made a new hardware design for FPGAs that takes advantage of this pattern to do the math faster and more efficiently. Their design works smoothly and uses little power, achieving very high speeds without extra overhead.
What this means in practice
- •For hardware accelerator designers: Implement faster and more power-efficient sparse matrix multiplication blocks specifically tailored for structured attention in transformer models.
- •For embedded system engineers: Build FPGA-based accelerators to speed up natural language processing tasks on edge devices using Longformer models with structured sparse attention.
Authors
Phillip Pramberger, Athanasios Tziouvaras, Shreejith Shanker, George Floros
Abstract
Sparse attention mechanisms have become increasingly important for transformer models processing long input sequences due to their lower computational and memory complexity compared to full self-attention. Longformer achieves this through a sliding-window attention mechanism that produces a structured banded sparse attention matrix. However, existing sparse transformer accelerators primarily target attention generation or unstructured sparsity, leaving sparse matrix--dense matrix multiplication (SpMM) for structured sparse attention largely unexplored. This paper presents a pipelined FPGA architecture for accelerating banded SpMM in Longformer. The proposed design exploits the predictable sparsity pattern of Longformer's attention matrix through a custom row-wise storage scheme with implicit indexing, eliminating the overhead of conventional sparse matrix formats while enabling regular memory accesses. The architecture employs parallel processing elements, pipelined adder trees, and a dual-path computation strategy to maximize throughput and hardware utilization. Implemented in Verilog and evaluated on an RFSoC platform using Vivado 2024.2, the accelerator sustains one complete dot-product result per clock cycle after an initial latency of 11 cycles while maintaining power consumption below 2.9 W. Operating at 100 MHz, the design achieves over 100 million dot-product outputs per second, demonstrating the effectiveness of directly exploiting structured sparsity for sparse transformer acceleration.