SketchSSM reduces state access to speed up transformer decoding
SketchSSM: Write to the Full State, Read from a Compact Sketch
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
Transformer models often need to read and update a large internal state when generating text, which slows down the process. The authors found a way to summarize that state using a set of fixed basis vectors, so the model only reads the full state once and then uses a smaller 'sketch' to answer future queries. Their method, called SketchSSM, reduces the data traffic needed to access the state by about ten times while keeping model accuracy close to normal. This leads to much faster text generation speeds on modern GPUs without sacrificing result quality.
What this means in practice
- •For machine learning engineers: Speed up transformer decoding by reducing state read traffic using compact sketches that approximate full state outputs.
- •For gpu system architects: Design GPU kernels optimized for SketchSSM to achieve higher transformer inference throughput on large batch sizes.
- •For real-time translation services: Use SketchSSM to improve decode speed in large-scale text generation models, enabling faster translations with minimal quality loss.$Commercial implications: Faster and more efficient translation models can be sold as premium services to enterprises requiring instant multilingual communication.
Authors
Omin Kwon, JoongWon Shin, Minseo Kim, Kurt Keutzer, Sehoon Kim, Jae W. Lee
Abstract
Hybrid-attention models replace most softmax attention layers with linear attention, reducing KV-cache growth and enabling larger decode batches where recurrent state access becomes a major bottleneck. ReplaySSM amortizes state updates by buffering keys and values, but each new query still requires a full-state read even though the state remains unchanged between state updates. We observe that low-rank state-weighted query approximation accurately preserves state-read outputs. Although future queries are unknown, the basis vectors used to approximate them can be fixed offline. Based on this observation, we introduce SketchSSM, which preserves full-state updates while approximating reads. At each state update, SketchSSM reads the full state once to precompute outputs for these basis vectors, storing them in a compact sketch. Each subsequent decode step combines the sketch vectors with query-dependent coefficients to reconstruct the output without a full-state read. Across four Mamba-2-, GDN-, and KDA-based models, SketchSSM reduces state-access traffic by approximately 10x while largely preserving average accuracy across four decode benchmarks and recall on four RULER retrieval tasks. On one NVIDIA B300, linear-attention kernel speedups over the standard vLLM baseline reach 7.78x, 5.22x, and 5.20x for Mamba-2, GDN, and KDA, respectively, with up to 2.64x higher decode throughput on Nemotron 3 Super.