DLite speeds up AI responses without storing full conversation history

DLite: History-free Speculative Decoding for Agentic AI

Computation and Language

Summary

When AI systems talk back and forth, keeping track of everything they said before can slow them down and make predictions less accurate. The authors found that this problem gets worse the longer the conversation goes on. They created DLite, a new way to guess the AI’s next words quickly without holding onto the entire chat history. This method uses parts of the AI’s internal thinking states and a special training approach to keep predictions reliable and fast. Their tests show DLite works better and faster than previous methods, especially for long conversations and interactive uses.

What this means in practice

  • For ai application developers: Cut response times in AI tools that process long conversations by using DLite’s history-free decoding technique to speed up multi-turn interactions.
  • For cloud service providers: Reduce computing load and system overhead during AI inference in services that involve continuous user interactions without storing full context histories.

Authors

Jiaming Xu, Hanzhen Wang, Tianlang Zhao, Jiewen Xiao, Chengze Yuan, Jiayi Pan, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai

Abstract

Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context, transforming an increasingly complex modeling problem into local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy. Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.