Large language models speed up by smartly switching copy and prediction
To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
Computation and Language
Summary
Large language models are smarter and faster when guessing words, but they often struggle between writing new content and copying repeated parts from what they already said. The authors studied this problem and created a system called SwitchSD that listens to the model’s own signals to tell when copying is really intended. This system switches between copying and writing new text more reliably, making the model work up to 15% faster. Their method improves both speed and accuracy by avoiding copying mistakes that slow things down.
What this means in practice
- •For machine learning engineers: Integrate SwitchSD to increase large language model inference speed by dynamically choosing between copying and generative drafting.
- •For cloud service providers: Deploy SwitchSD-enhanced language models to reduce hardware costs and improve response times in text generation services.$Commercial implications: Enables faster and more cost-efficient text generation APIs with adaptive decoding, appealing to businesses running large-scale AI services.
Authors
Roy Eisenstadt, Ido Cohen, Edo Cohen-Karlik, Lior Wolf, Itamar Zimerman
Abstract
Speculative Decoding (SD) has significantly accelerated Large Language Model (LLM) inference, yet existing approaches face a fundamental tradeoff between two drafting strategies: neural drafting and context-based copying. Neural drafts (e.g., EAGLE3) provide robust performance across diverse text settings, while copy-based methods achieve higher speedups in copy-intensive regimes by generating candidates faster and exploiting long repetition spans for near-perfect speculation. We analyze existing copy-based methods and find that they are prone to accidental repetitions where surface-level n-gram overlap does not reflect a structural intent to copy, leading to false-positive triggers that ultimately degrade throughput. We introduce SwitchSD, an adaptive framework that treats copying as a latent control signal of the LLM. By training lightweight probes on the target model's internal representations, SwitchSD identifies genuine copy-intent with high precision (AUC > 0.99). This allows the system to dynamically switch between neural drafting (e.g., EAGLE) and context-based copying. Our results across Llama and Qwen families demonstrate throughput gains of up to 15% over state-of-the-art baselines like EAGLE3, effectively turning copying from a noisy heuristic into a principled, model-aware decoding regime.