Transformer attention made faster on long-vector processors
Reorienting the Vector Axis: Restructuring FlashAttention for Long-Vector Processors
Hardware ArchitectureDistributed, Parallel, and Cluster ComputingPerformance
Summary
Transformer models use attention, which becomes slow on long sequences because the work grows quickly. The authors studied a kind of fast attention method called FlashAttention but on special processors with very long vector registers, different from the usual GPUs. They found the original method wasn’t using the long vectors well, so they redesigned how data moves through the process to better fit the hardware. Their changes almost doubled the computational speed on the test hardware without changing the calculation results.
What this means in practice
- •For cpu hardware engineers: Design more efficient attention kernels for long-vector CPUs to speed up Transformer inference and training workloads.
- •For fpga system designers: Optimize Transformer attention implementations on FPGA platforms by adopting data layouts that align better with vector registers.
Authors
Emmanuel Irabor, Marc Casas
Abstract
A Transformer is built around attention, whose cost grows with the square of the sequence length and dominates the layer on long sequences. FlashAttention made that cost manageable by tiling the attention matrix and updating the softmax as it goes, but almost all work since has targeted GPUs. On long-vector processors one vector register can hold up to 512 single-precision numbers. The savings of a vector instruction are proportional to how much of the register it uses, so the choice of which loop to vectorize is a first-order decision. We port PyTorch's CPU FlashAttention kernel to a RISC-V CPU with 2KB vector registers and find that it reaches 16.2% of peak floating-point throughput. The kernel walks each tile one query row at a time: the product against V is vectorized along the 64-element head dimension, the softmax's per-row maximum and sum are reductions that collapse a full register to a scalar, and the software exponential's polynomial coefficients compete with tile data for vector registers and cause spilling. We propose two techniques. The first transposes the score tile and the running output so that queries lie along the contiguous axis. The algorithm and its values are unchanged. A single parameter, the number of queries in a tile, now sets the vector length of every part of the kernel and can be matched to the machine. Both reductions also disappear. The second moves the polynomial's coefficients into scalar registers, removing the spills and cutting the softmax's vector memory instructions by 78.7%. On silicon the two together cut whole-kernel cycles by 1.84 to 2.08x at a head dimension of 64, raising floating-point utilization from 16.2% to 29.7% of peak. An FPGA of the same design ranks the changes identically and scores them higher, at 2.71x.