AttnFuse speeds up transformer attention with fused GPU kernels
AttnFuse: A Composable DSL for Compiling Attentions to Fused GPU Kernels
Machine Learning
Summary
Transformer models use attention operations that take up most of their computing power. The authors created AttnFuse, a small language that helps programmers combine all parts of attention computation, including special position encoding like RoPE, into one fast GPU program. This approach runs faster than recent alternatives and comes close to manually optimized code. They also found rules about when to combine certain steps depending on the GPU’s hardware. AttnFuse shows it's possible to balance flexibility for researchers with speed for real use.
What this means in practice
- •For machine learning engineers: Run custom attention variants efficiently on GPUs for improved transformer training and inference.
- •For graphics hardware designers: Use insights on compute-to-bandwidth tradeoffs to optimize GPU architectures for transformer workloads.
Authors
Varun Kumar Dasoju, Tian Zhao
Abstract
Modern AI systems are built on the Transformer architecture, whose core operation, attention, accounts for the majority of computation and memory cost. Researchers continually propose new attention variants to improve quality, efficiency, or context length, but each variant currently requires expert-written GPU code to run at usable speeds. PyTorch's recent flex\_attention lets researchers describe custom attention patterns in Python and compile them to fused kernels, but its design is limited to modifications applied after the central matrix multiplication, excluding Rotary Position Embedding (RoPE), the positional encoding used by every major LLM. We introduce AttnFuse, a small DSL for attention that makes pre-multiplication transformations like RoPE first-class operations. Researchers compose ten high-level building blocks to describe a variant, and AttnFuse's compiler emits a single fused GPU kernel for the entire computation. On an RTX 3090, AttnFuse achieves a 2.10$\times$ speedup over flex\_attention on the RoPE+causal pattern. On an H100, it runs a full Llama-3-8B training step within 5\% of PyTorch's hand-tuned backend. Our investigation reveals the Rotation Calculus: whether to fuse RoPE or apply it separately depends on the GPU's compute-to-bandwidth ratio, with a derived crossover that matches measurement. AttnFuse demonstrates that a small, attention-specific compiler can close the gap between flexible research code and production kernels.