ReWY speeds up gradient computation for delta-rule attention models
ReWY: Faster Gradient Computation for Delta-Rule Attention Family
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
Some language models use a technique called delta-rule attention that needs a lot of calculations to learn. The authors created a new way called ReWY to simplify part of these calculations, making the process faster without changing how the models learn or their results. They tested this on different GPUs and hardware accelerators, showing ReWY can speed up the backward pass computations by about 1.1 to nearly 3 times. This improvement helps run models that use delta-rule attention more efficiently on existing hardware.
What this means in practice
- •For ai engineers: Accelerate training of language models using delta-rule linear attention by reducing backward pass computations on GPUs.
- •For hardware accelerator developers: Incorporate ReWY's reduced computation graph into accelerator software stacks to improve support for efficient delta-rule attention layers.
Authors
Yi Zhang, Zhixin Zhao, Weiwei Sun, Yitao Hu, Sheng Chen, Keqiu Li
Abstract
Delta-rule linear attention underlies language models such as Kimi Linear and Qwen3-Next. Chunkwise implementations expose matrix parallelism, but their backward passes compute unnecessary auxiliary adjoints. We present ReWY, an algebraic reformulation that reduces this work before optimizing accelerator execution. Differentiating the chunk correction as one triangular residual system eliminates the inverse adjoint and reduces seven local WY matrix products to three. Two complementary transformations expose further reuse: centered matrix contractions replace pairwise vector-gate processing, and scalar-gate reductions reuse key-gradient products instead of forming a key Gram matrix. These identities simplify backward computation for DeltaNet, GDN, KDA, and GDN2 while preserving the recurrence and gradients in exact arithmetic, without changing the architecture or training objective. We implement the reduced graph on GPUs and Ascend NPUs. Under matched recomputation and full-saving policies, precision-qualified six-shape comparisons with frozen FLA references yield 1.15-1.81$\times$ geometric-mean complete low-level backward speedups on NVIDIA GPUs. All four operators achieve 1.18-1.70$\times$ on H100 and H200 while retaining original FP32 working matrices. A separate public-autograd evaluation on Ascend 950DT, including gradient conversions, yields 1.09-2.89$\times$ under available matched policies. Paired ablations establish further ReWY gains after optimizing the complementary contractions. Complete-gradient checks validate the recorded operator cases; supplementary full-model backward measurements assess integration into community FLA models and a GDN2 layer port. Code and reproduction materials: https://github.com/flashserve/ReWY.