CopyLoop speeds up transformer decoding by copying context and shallow checks
CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening
Computation and LanguageArtificial Intelligence
Summary
Generating text with some AI models can be slow because they process each word step-by-step and at full depth. The authors introduce CopyLoop, a way to speed this up by guessing multiple words at once from repeated parts of the text and quickly checking if those guesses seem right. This approach works well across tasks like coding and math, cutting down the time it takes to produce text by nearly three to ten times for some models. It also only needs small adjustments to the AI model and works better on longer pieces of text where parts can be reused.
What this means in practice
- •For software developers: Generate longer code snippets faster by using context-based multi-token proposals that reduce computation time during AI-assisted coding.
- •For math tutoring platforms: Accelerate multi-step math problem solving in AI tutors by quickly filtering and verifying reasoning steps with minimal computational overhead.
Authors
Wenyu Du, Stephen Chung
Abstract
Looped Transformers increase computational capacity through the repeated application of shared blocks, but their recurrent depth makes conventional full-depth decoding expensive. While existing speculative pipelines accelerate generation by processing consecutive token positions at varying depths, they are typically limited by sequential decoding overheads. We introduce CopyLoop, a training-free decoding algorithm that combines low-cost context-copy proposals with shallow screening and continuous recurrent scheduling. Rather than step-by-step generation, CopyLoop proposes multi-token suffixes from repeated context. It then evaluates them early at a shallow recurrent depth to efficiently filter out incorrect branches, processing newer blocks concurrently while earlier blocks are still progressing toward full-depth verification. We evaluate CopyLoop across six model checkpoints from the Ouro, Raven, and Huginn families, covering code generation and multi-step mathematical reasoning. CopyLoop achieves decoding speedups of 2.98x--10.70x over full-depth autoregressive decoding on Base models and 2.74x--3.52x on Thinking models, consistently outperforming or closely matching all competing baselines across evaluation tasks. Furthermore, ablation studies confirm a strong synergy between shallow screening and continuous scheduling, while our analyses show that CopyLoop requires only lightweight model calibration and yields greater speedups on longer generations with reusable contexts.