Papers for

real-time translation services

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

SketchSSM reduces state access to speed up transformer decoding

SketchSSM: Write to the Full State, Read from a Compact Sketch

Abstract: Hybrid-attention models replace most softmax attention layers with linear attention, reducing KV-cache growth and enabling larger decode batches where recurrent state access becomes a major bottleneck. ReplaySSM amortizes state updates by buffering keys and values, but each new query still requires a full-state read even though the state remains unchanged between state updates. We observe that low-rank state-weighted query approximation accurately preserves state-read outputs. Although future queries are unknown, the basis vectors used to approximate them can be fixed offline. Based on this observation, we introduce SketchSSM, which preserves full-state updates while approximating reads. At each state update, SketchSSM reads the full state once to precompute outputs for these basis vectors, storing them in a compact sketch. Each subsequent decode step combines the sketch vectors with query-dependent coefficients to reconstruct the output without a full-state read. Across four Mamba-2-, GDN-, and KDA-based models, SketchSSM reduces state-access traffic by approximately 10x while largely preserving average accuracy across four decode benchmarks and recall on four RULER retrieval tasks. On one NVIDIA B300, linear-attention kernel speedups over the standard vLLM baseline reach 7.78x, 5.22x, and 5.20x for Mamba-2, GDN, and KDA, respectively, with up to 2.64x higher decode throughput on Nemotron 3 Super.

Sun 27 SeptMachine LearningDistributed, Parallel, and Cluster Computing
The gist
Transformer models often need to read and update a large internal state when generating text, which slows down the process. The authors found a way to summarize that state using a set of fixed basis vectors, so the model only reads the full state once and then uses a smaller 'sketch' to answer future queries. Their method, called SketchSSM, reduces the data traffic needed to access the state by about ten times while keeping model accuracy close to normal. This leads to much faster text generation speeds on modern GPUs without sacrificing result quality.
Open → 2609.33051v1