Transformer model speeds up generation by changing matrix math
Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers
Machine LearningPerformance
Summary
Transformers use math that multiplies big matrices to understand and generate text. The paper shows you can change the way this multiplication works to make processing faster without changing the number of parameters. The authors tested this new math in a language model training with the same size but found it was faster while producing slightly lower-quality results. This idea could lead to more efficient models if improved further.
What this means in practice
- •For machine learning engineers: Accelerate text generation in Transformer models by using associative algebra-based matrix products to increase throughput without changing model size.
- •For hardware accelerator designers: Design GPU-friendly matrix multiplication methods that optimize Transformer feed-forward layers for faster inference using sparse interaction tables.
Authors
Ilya Koziev, Ivan Oseledets
Abstract
Fast matrix multiplication algorithms keep the product fixed and search for a cheaper way to evaluate it. We instead ask whether a Transformer's learned projections can use a different, cheaper product altogether. Building on an associative-algebra construction that replaces ordinary matrix multiplication with a sparser interaction table over the same weight blocks, we construct a family with quadratic arithmetic in the matrix dimension when the physical block size remains fixed, and derive finite-shape constraints for GPU execution. The construction is provably optimal for its bilinear rank by the Alder--Strassen bound and can be realized as row-typed rectangular projections compatible with causal masking and KV-cached decoding. We provide an empirical test of this approach by training two approximately 110M-parameter decoder-only Transformer LMs from the same recipe and 12.3B-token budget, differing only in their feed-forward layer: one uses ordinary dense matrix multiplication and the other uses the associative-algebra product. Across four prompt domains, the algebraic model achieves a 6.2--7.8\% increase in end-to-end generation throughput, while obtaining lower scores on all three reported downstream metrics. We treat these results as a feasibility and trainability check for the proposed approach at small scale, leaving further investigation to future work.